S01E15 A phase transition is brewing (season finale)
Phase transitions, entropy, sociotechnical systems, the second law of thermodynamics and the third law of sci-fi
Housekeeping
I am going to make some changes to the newsletter format and frequency. The combination of weekly long-form content and curated links is a bit much. I suspect this is true both for the writer and the reader. Starting next week, I will alternate between curation and essays.
Today, I wrap up this first season. While I have only scraped the surface of the body of knowledge that is complexity theory, I feel that I have done enough to explain why we should question our default approach to organization. At the end of this episode, I introduce what’s in store for season two.
“No problem stays solved in a dynamic environment.”
—Russell L. Ackoff
S01E15 A phase transition is brewing
In last week’s episode, I investigated how complex systems emerge from simpler systems at the edge of chaos. It became clear that self-organization is a fundamental feature of complex, adaptive systems.
Self-organization and interaction with the environment allow systems to adapt and evolve. In a reinforcing feedback loop, the newly emerged structure increases the scope of cooperation and competition, enabling yet more self-organization and evolution:
Examples of this feedback loop can be found in the natural sciences (physics, biology, astronomy…) and social sciences (psychology, economics, sociology…). I’ll give examples from different domains, and try to shed some light on the enigmatic link between self-organization and emergence - the phase transition.
In early human history, when hunter-gatherers abandoned their nomadic existence to self-organize into settlements, this set off a phase transition. New societal structures started to emerge:
increased food production and food surpluses led to larger populations
larger populations gave rise to new social hierarchies and classes
social technologies (specialization of labor, law, writing…) enabled new ways to compete and collaborate
social complexification led to the invention of new tools and technologies (irrigation, pottery, metalworking…)
The agricultural revolution introduced new non-zero-sum games and heralded an increasingly sophisticated society. While there have been historical periods of decline (e.g., after the fall of the Roman Empire), it is fair to say that directionally we are evolving toward more complex societies.
Agriculture was the first of many societal phase transitions. Historians usually refer to them as revolutions, like the scientific, industrial, urbanization, and information revolutions.
What do we know about phase transitions?
Phase transitions are an endless source of fascination for scientists. Entirely new properties spring to life when systems transition from one phase to another (e.g. from ice to water, or from nomadic culture to agriculture). Much remains to be discovered - understanding these evolutions is one of the fundamental questions that complexity theory addresses.
Some patterns do emerge. Repeated perturbations in a system - the result of the conflicting forces of competition and collaboration - result in a “frustrated state” and cause increasing instability in the system’s equilibrium. Instability impacts the anchors of that system and sets off a phase transition until a new equilibrium is achieved.
In thermodynamics, like in the liquid-gas phase change of boiling water, these transitions are relatively well understood (or so my physicist friends assure me). Our understanding of these transitions becomes fuzzier when we explore human and societal interactions, as do the boundaries between states.
Phase transitions can be observed at different levels of analysis (organic, molecular, organic, psychological, social…). For example, we’ve all witnessed the phase change when laughter suddenly lightens the mood in a roomful of people. We can see them after an outstanding music performance when applause transitions into a standing ovation.
I started my career in traditional project management, and I have been at the center of more than one phase transition of truth. This phenomenon occurs when the project indicators in a spreadsheet suddenly go from orange to red, catching managers off guard (but rarely the developers).
This is an interesting aspect of phase transitions. The emerging changes often aren’t visible on the surface. In an example from physics, when ice melts it reaches a plateau where heat is continually added to the system, but it takes time for the phase transition to reach the liquid state.
The above image might trigger flashbacks to physics classes. In fact, there is a metaphysical question I have to address at this point. One of the most fundamental laws of physics states that in closed systems, the total entropy increases over time:
The law that entropy increases—the Second Law of Thermodynamics—holds, I think, the supreme position among the laws of Nature.
How, then, can we reconcile increasing disorder (entropy) with increasing sophistication (complexity)?
I will address this question with my trademark weapons: clarification through metaphors, triangulation of sources smarter than me, and a well-chosen Youtube video. I would also like to throw down the gauntlet to my more math- and physics-oriented friend Maarten Mortier in the hope that he takes a stab at the topic.
What is entropy, and why does it matter?
Entropy refers to the natural tendency of things to lose order over time. It is the reason why abandoned houses disintegrate into the environment, why cars rust, and why software systems tend to evolve into big balls of mud.
In theory, with infinite monkeys hitting keys on typewriters, one of them could reproduce the collected works of Shakespeare. In our entropic reality, the chances are vanishingly small. The second law is a matter of probability - there are many disordered states and few ordered ones.
Entropy is a scary subject for non-physicists. For an eye-opening, yet accessible introduction to entropy, emergence, and complexity, I warmly recommend reading Sean Carroll’s book The Big Picture: On the Origins of Life, Meaning, and the Universe Itself.
Sean Carroll is a professor at Johns Hopkins and the Santa Fe Institute (the mecca of complexity science). In his book, Carroll sheds light on the surprising role of entropy as the driving force behind complexity. While this sounds like a paradox, he explains how entropy, the very concept that will eventually lead to a boring and lifeless universe, is a necessary requirement for emerging complexity:
While there is no escaping the second law of thermodynamics, we can fight back against entropy. In order to successfully curb entropy and create stability, structure, and simplicity, we need to expend energy. Order requires effort.
Physicist Sharon Glotzer has conducted groundbreaking research on this topic, specifically on the principles that govern how macroscopic properties emerge from microscopic interactions. In exploring this question, she connected entropy with self-organization:
“We typically think entropy means disorder, and so a disordered structure would have more entropy than an ordered structure. That can be true under certain circumstances, but it’s not always true, and in these cases, it’s not. I prefer to think of entropy as related to options: The more options a system of particles has to arrange itself, the higher the entropy. In certain circumstances, it’s possible for a system to have more options – more possible arrangements – of its building blocks if the system is ordered.”
She further explains that entropy sets particles to “wiggling.” As it turns out, particles naturally maximize their individual space for movement. On their entropic quest for wiggle room, particles order themselves into more complex - and more ordered - states.
This is a good moment to reassure my esteemed readers. I know that you signed up for digital transformation wisdom, not for the wiggling of theoretical particles. I will not venture further on the ice of complexity (lest I fall through it), but I felt the need to show the reader that complexity theory is on solid scientific footing.
“Theories are like scaffolding: they are not the house, but you cannot build the house without them.” — Constance Fenimore Woolson
Entropy may be an abstract concept, but organizations deal with it all the time. Understanding concepts like entropy, even at a high level, can help inform very practical decisions in business strategy or software architecture.
Disorder and variation form the surface area for innovation. The companies that successfully wrangle entropy will outmaneuver the companies that merely follow a script, incapable of trying new things (see: live players vs dead players).
What’s in store for season two?
With the theoretical groundwork in place, I want to explore the ‘how’ in more detail. How can we apply complexity theory in management practice?
When all you have is a hammer, everything looks like a nail.
Over the next dozen or so episodes, I will explore the expansive and varied toolbox of sociotechnical systems management. There are many useful techniques, tools, and frameworks out there. We just need to put in the effort of learning how and when to use them.
Sociotechnical?
So far, I’ve referred to organizations as complex, adaptive systems. As we move from theory to practice, it is more helpful to think of companies as sociotechnical systems. In the words of Dave Snowden: people aren’t ants.
The sociotechnical frame implies that organizational systems can only be understood and improved if both social and technical aspects are considered and treated as interdependent parts of a complex system. Most approaches to digital transformation start from an overly narrow perspective (most often technological).
The sociotechnical systems approach is rooted in complexity theory and systems thinking - it takes a broad view. Concepts like feedback, modeling, self-criticality, emergence, and entropy can help account for the social and technical drivers of an organization (as well as the interconnectedness between its social and technical attributes).
What to take away from season one?
For all this writing on complexity, the conclusion is a simple one: we are in the midst of a phase transition. Organizational systems have entered into a frustrated state.
Under a thin veneer of engagement, most employees are larping their jobs. Dilbert cartoons are funny because they are true. Actual motivation, cooperation, or innovation are scarce in the corporate universe (exceptions exist, there is an avant-garde cohort that is taking a different approach).
I made the point that most organizations don’t live up to their potential because they run deprecated managerial software. Traditional operating models are a legacy from the industrial era. We have been trying to manage complex systems through convoluted control structures, rooted in a mechanistic worldview.
I believe that complexity theory contains the seeds for a new management theory, one that enables sociotechnical systems to thrive in the variety of the information era. Instead of focusing on control and uniformity, companies will learn to embrace flux, change, and the forming and dissolving of patterns.
Reductionist methodologies, checklists, and generic scripts will give way to the science of uncertainty. The new tenets are:
Take more, but smaller steps. This allows organizations to:
experiment
learn faster
minimize risks
reason about cause and effects
Normalize change and think probabilistically. Taking a linear approach to causality is futile: even if we know all the inputs, we cannot predict how a system will behave.
Improve your understanding of the system through mapping or modeling, but appreciate that the system reacts to the model.
Focus not just on the components but also on the relations between the components (both social and technical).
Create structures that let people work their magic. The recent leap forward of AI has reaffirmed Clarke’s third law: sufficiently advanced technology is indistinguishable from magic. How to make it as a leader in the magical era?
Do not try design outcomes. Instead, reason about the rules, habits, and incentives that we allow to shape the system (“governance”).
Systems change over time, and change is driven by numerous iterations of very simple rules. To change the behavior, start with the rules.
Treat employees as wizards, not as serfs.
🤓 Further reading
The Big Picture: On the Origins of Life, Meaning, and the Universe Itself - Sean Carroll