Historical Progression of Complexity, Networks and Hierarchy

From P2P Foundation
Jump to navigation Jump to search

Great graph from the article Complexity Rising, which is an overview of the relationship between complexity an hierarchy. The graph shows how hierarchical relations are being replaced by lateral relations.

This is also a key article to read to understand the trend towards distributed networks.

URL = http://necsi.org/figs/histprog.jpg

Full article at http://necsi.org/projects/yaneer/Civilization.html


How Hierarchy hinders Complexity

From the article:


Excerpted from the article at http://necsi.org/projects/yaneer/Civilization.html


In human hierarchies the collective behavior must be simple enough to be represented by a single human being.


"The history of human civilization reflects a progressive increase in the complexity of large scale behaviors. Early civilizations introduced a few relatively simple large scale behaviors by use of many individuals (slaves or soldiers) performing the same repetitive task. Progressive specialization with coordination increased the complexity of large scale behaviors. The industrial revolution accelerated this process which continues till today. When the complexity of collective behaviors increases beyond that of an individual human being then hierarchical controls become ineffective."


"Distributed control over collective behaviors can result in larger complexity of the collective behavior than the behavior of any single individual. Networks are also quite distinct from independent individuals. Networks require that coordination of the behavior of groups of individuals are achieved by mutual influences."

Main thesis:

"Hierarchical organizations are designed to impose correlations in human behavior primarily through the influence of the hierarchical control structure. In an ideal hierarchy all influences/communications between two "workers" must travel through a common manager. As the complexity of collective behavior increases, the number of independent influences increases, and a manager becomes unable to process/communicate all of them. Increasing the number of managers and decreasing the branching ratio (the number of individuals supervised by one manager) helps. However, this strategy is defeated when the complexity of collective behavior increases beyond the complexity of an individual. Networks allowing more direct lateral interactions do not suffer from this limitation" (http://necsi.org/projects/yaneer/Civilization.html)

Detailed argument:

"This section focuses on internal interactions that at any one time give rise to collective behaviors. In human organizations coordination occurs because individuals influence each others' behavior. The influence is often called control. It is not necessarily coercive control, though coercion may be an aspect of control. The objective of this section is to understand the relationship between control structure and the complexity of collective behavior.

Real human hierarchical organizations are not strict hierarchies, they contain lateral interactions that enable control to bypass the hierarchy. However, by focusing on an idealized control hierarchy it is possible to understand the nature of this structure. Such a focus will help in understanding the nature of dictatorships and hierarchical corporationsthe relationship between these control structures and complex collective behavior. In an idealized hierarchy all communication, and thus coordination of activities, is performed through the hierarchy.

To concretize the discussion, consider two paradigmatic examples: military force and factory production. Conventional military behavior is closer to our discussion of coherent behavior. Similar to coherent motion, in the military the behavior of an individual is simplified to a limited set of patterns. The behavior patternssuch as long marcheshave a high degree of repetition and thus can have impact on a large scale. Then, many individuals perform the large-scale behaviors coherently. While this model continues to apply to some examples of modern military activity, the diversity of actions of a modern military makes this model better suited to understanding ancient armiesRoman legions, or even U. S. Civil War armies.

While the actions of the military are designed to have impact on a large scale, they must still be performed in response to specific external conditions. As the conditions change, the actions must also be changed. There is need for a response mechanism that involves communications that can control the collective behaviors. Such a response generally involves direct action by the control hierarchy.

A conventional industrial production line also simplifies the behavior of an individual. Each individual performs a particular repetitive task. The effect of many individuals performing repetitive tasks results in a large number of copies of a particular product. This repetition increases the scale of impact of an individual's behavior. However, unlike coherent behavior, the behavior of different individuals is not the same. Instead, the activities of the individual are coordinated to those of othersthe coordination exists so that the larger-scale behavior can arise. The coordination means that the behaviors of different individuals, while not the same, are related to each other. When compared to the coherent motion, this increases the complexity and decreases the scale, but much less so than would be the case for fully independent individuals.

The need to ensure coordination of different individuals when the collective actions being performed have an inherently higher complexity increases the demands upon the control hierarchy. In particular, it is significant that the behaviors of all parts of a production line must be coordinated, even though actions being performed are different.

The similarities and differences between the factory and the military models are relevant to an understanding of the role of hierarchical control. A military force, a corporation, or a country have behaviors on various scales. At larger scales, many of the details of the behavior of individuals are not apparent. Intuitively, a control hierarchy is designed to enable a single individual (the controller) to control the collective behavior, but not directly the behavior of each individual. Indeed, the behavior of an individual need not be known to the controller. What is necessary is a mechanism for ensuring that control over the collective behavior be translated into controls that are exercised over each individual. This is the purpose of the control hierarchy.

A hierarchy, however, imposes a limitation on the degree of complexity of collective behaviors of the system. This can be understood by considering more carefully the processes of coordination. The hierarchy is responsible for ensuring coordination of various parts of the system. Lower levels of the hierarchy are responsible for locally coordinating smaller parts of the system and higher levels of the hierarchy are responsible for coordinating the larger parts of the system. At each level of the hierarchy the actions to be coordinated must be transferred through the controller. Thus, the controller's behavior must itself reflect all of the impacts that different parts of the system have on other parts of the system. This implies that the collective actions of the system in which the parts of the system affect other parts of the system must be no more complex than the controller. In human hierarchies the collective behavior must be simple enough to be represented by a single human being.

In summary, the complexity of the collective behavior must be smaller than the complexity of the controlling individual. A group of individuals whose collective behavior is controlled by a single individual cannot behave in a more complex way than the individual who is exercising the control. Hierarchical control structures are symptomatic of collective behavior that is no more complex than one individual. Comparing an individual human being with the hierarchy as an entirety, the hierarchy amplifies the scale of the behavior of an individual, but does not increase its complexity.

The existence of lateral influences counters these conclusions with respect to real human organizations. These lateral controls are similar to the conceptual networks that are used to model the interactions between neurons in the brain. Distributed control over collective behaviors can result in larger complexity of the collective behavior than the behavior of any single individual. Networks are also quite distinct from independent individuals. Networks require that coordination of the behavior of groups of individuals are achieved by mutual influences." (http://necsi.org/projects/yaneer/Civilization.html)

The historical development of complexity

From the article:


Excerpted from the article at http://necsi.org/projects/yaneer/Civilization.html

"In recent years human organizations that emphasized central control have changed or given way to other structures with greater distribution of control. This includes political organizationsthe systematic conversion of dictatorships in Central and South America to more democratic systems, the fragmentation of the soviet bloc and replacement of government controlled economies in communist countries with market based economiesand the restructuring of hierarchical corporations in western economies to involve decision teams and process based managerial strategies. Many of these changes result in systems where collective behaviors arise from partially independent subgroups of the system and lateral "networked" influences. Even when control hierarchies continue to exist, the lateral interactions through group decision making processes have become more prominent. To understand this more fully, consider the history of civilization and the complexity of environmental demands upon each civilization and the individuals that comprise it. The progressive historical increase of complexity means that organizations that do not change do not survive. This is descriptive of the nature of the transition that is under way. The complexity of demands upon collective human systems have recently become larger than an individual human being. Once this is true, hierarchical mechanisms are no longer able to impose the necessary coordination of individual behaviors. Instead, interactions characteristic of networks are necessary.

In a review of history, the development of hierarchies can be seen to enable progressively more complex behaviors. Two factors are important, progressively smaller branching ratios and lateral interactions. Both will be described below. There are also two complementary aspects to this development, complexity at the scale of the individual and complexity at the scale of the collective. In general, these complexities are not directly related. In the context of a control hierarchy, however, the complexity of individual behaviors increases with increasing complexity of collective behavior. The complexity/diversity of individual behaviors does not directly explain the difficulties experienced by hierarchies. The complexity of collective behaviors does explain the difficulties experienced by control hierarchies, since controlling these behaviors is the role of central control.

From earliest recorded history until the fall of the Roman empire, empires replaced various smaller kingdoms that had developed during a process of consolidation of yet smaller associations of human beings. The degree of control exercised in these systems varied, but the progression toward larger more centrally controlled systems is apparent. As per our discussion of the difference between independent individuals and coherent behaviors, this process was driven by military force.

Indeed, during the time of ancient empires, large-scale human systems executed relatively simple behaviors, and individuals performed relatively simple individual tasks that were repeated by many individuals over time to have a large-scale effect. This observation applies to soldier armies, as well as slaves working in agriculture, mines or construction. The scale of ancient empires controlled by large armies, as well as the scale of major projects of construction would be impressive if performed today. The scale of activity was possible, without modern sources of energy and technology, because of the large number of individuals involved. However, the nature of the activity was simple enough that one individual could direct a large number of individuals. Thus, hierarchies had a large branching ratioeach controller was in charge of a large number of individuals.

As time progressed, the behavior of individuals diversified as did the collective tasks they performed. The increasing diversity of individual behaviors implies an increase in the complexity of the entire system viewed at the scale of the individual. Consequently, this required reducing the branching ratio by adding layers of management that served to exercise local control. As viewed by higher levels of management, each layer simplified the behavior to the point where an individual could control it. The hierarchy acts as a mechanism for communication of information to and from management. The role is also a filtering one, where the amount of information is reduced on the way up. Conversely, commands from the top are elaborated (made more complex) on the way down the hierarchy. As the collective behavioral complexity at the scale of an individual increases, the branching ratio of the control structure becomes smaller and smaller so that fewer individuals are directed by a single manager, and the number of layers of management increases. The formation of such branching structures allows an inherently more complex local behavior of the individuals, and a larger complexity of the collective behavior as well.

The most dramatic increases in the complexity of organizational behavior followed the industrial revolution. The use of new energy sources and automation enabled larger scale behavior in and of itself. This, in turn, enabled higher complexity behaviors of human systems because the amplification of the behavior to a larger scale can be accomplished by the use of energy rather than by task repetition.

At the point at which the collective complexity reaches the complexity of an individual, the process of complexity increase encounters the limitations of hierarchical structures. Hierarchical structures are not able to provide a higher complexity and must give way to structures that are dominated by lateral interactions. A hierarchy serves to create correlations in the behavior of individuals that are similar in many ways to the behavior of a network. The hierarchy serves as a kind of scaffolding. At the transition point, it becomes impossible to exercise control, so the management effectively becomes divorced from the functional aspects of the system. Lateral interactions that replace the control function have been present in hierarchical structures, however, they become necessary when the hierarchical control structure fails due to the high complexity of collective behavior. The greater the dependence of a system on the hierarchy, the more dramatic the changes that then take place.

The lateral interactions achieve the correlations in behavior that were previously created by management. As such mechanisms are introduced, layers of management can be removed. Over the course of the transition, the hierarchy exercises control over progressively more limited aspects of the system behavior. Some of the behavior patterns that were established through the control hierarchy may continue to be effective; others cannot be since an increase in system complexity must come about through changes in behavior. Among these changes are the coordination mechanisms themselves, which must be modified. It could be argued that this picture describes much of the dynamics of modern corporations. Upper levels of management have turned to controlling fiscal rather than production aspects of the corporation. In recent years, corporate downsizing has often been primarily at the expense of the middle management, resulting in a reduction of payroll and little change in production. Hierarchical control has been replaced by decision teams introduced by corporate restructuring; and the reengineering of corporations has focused on the development of task related processes that do not depend on hierarchical control.

Using this argument it is straightforward to understand why control structures ranging from communism to corporate hierarchies could not perform the control tasks required of them in recent times. As long as the activities of individuals were uniform and could be simply describedfor example, soldiers marching in a row, or manufacturing workers producing a single product by a set of repetitive and simple activities (pasting eyes on a doll, screwing in bolts)control could be exercised. The individual's activities can be specified once for a long period of time, and the overall behavior of the collective could be simply described. The collective behavior was simple; it could be summarized using a description of a simple product and the rate of its production. In contrast, central control cannot function when activities of individuals produce many products whose description is complex; when production lines use a large number of steps to manufacture many different products; when the products vary rapidly in time; and the markets change rapidly because they themselves are formed of individuals with different and rapidly changing activities.

It is useful to distinguish networks that coordinate human activity from markets that coordinate resource allocation. Markets are a distinct type of system that also results in an emergent collective behavior based upon the independent actions of many individuals. Markets such as the stock exchanges or commodity markets coordinate the allocation of resources (capital, labor and materials) according to the dynamically changing value of their use in different applications. Markets function through the actions of many agents (individuals, corporations and aggregate funds). Each agent acts according to a limited set of local objectives, while the collective behavior can coordinate the transfer of resources across many uses. Markets are distinct from networks in that they assume that the interactions among all agents in regard to a single resource can be summarized by a single time-dependent variable which is the value of the relevant resource.

In conclusion, the implication of the disappearance or dramatic changes in centrally controlled human organizations is that the behaviors of collections of human beings do not simplify sufficiently to be controlled by individuals. Instead of progressive simplification from an individual to larger and larger collections of individuals, we have the oppositean increasing complexity that is tied to an increasing complexity of the demands of the environment. This makes it impossible for an individual to effectively control collective behaviors. While specific individuals have been faulted for management errors that have led to corporate failures, the analysis performed here suggests that it is inevitable for management to make errors under these circumstances.

Finally, from an academic point of view, for those interested in developing an understanding of the political, social or economic behavior of the human civilization or its various parts, there are several important consequences. The high collective complexity implies that as individuals we are unable to fully understand the collective behavior. This does not mean that insights and partial understandings are impossible. However, the existence of many different scales of behavior in a complex system implies that two traditional approaches to modeling or considering such systems cannot be effective. The first assumes that the collective behavior can be understood solely from large scale interactions; specifically, a description of the interactions between nations. The second assumes that the collective behavior can be understood by decomposing the system into its smallest elements and developing models based upon individual behavior. A complete specification of each of the physical components of a system would describe also the collective system behavior, however, such a complete specification is impossible. Mapping or simulating all individual behaviors is ineffective as an approach to gaining understanding. This reductionist view, dominating much of the scientific thought, does not take into consideration the significance of large scale correlations essential to the complex collective behaviors we would like to understand. Effective models must build descriptions that account both for the many scales of behavior of a system and the interplay between environmental and system properties. In addition, it is the dynamic behavior patterns of the system that must be the focus of the understanding."

More Information

See the entry on Hierarchy