My Bannable Opinion: The Technological Singularly is Us
- Trevor Alexander Nestor
- May 19
- 21 min read
A presentation on cybernetics theory and the late-stage dynamics of complex societies, with
implications for how we should interpret "the technological singularity."
Video presentation: https://youtu.be/HhQtzPtoncE
Introduction
I want to present on cybernetics theory as it relates to the current situation in the United States
as an empire. Many of us have noticed that things have seemed unstable recently, with cultural,
economic, geopolitical, and social disruptions all converging at once. It can feel difficult to grasp
what is going on, like the world is collapsing around us, and harder still to make sense of it. My
hope is that this framework helps clarify the underlying dynamics. These opinions are for some reason just not allowed on mainstream platforms like Reddit or LessWrong (even when well cited or provably not generated by AI which you can review in natural cadence is directly from me in my video presentation where most is a direct transcript from the presentation).
Cybernetics is the area of research concerned with modeling and running societies by borrowing
from physics. Just as the laws of nature govern how physical systems behave, it should be
possible to apply those laws rigorously to the behavior of groups of people, because groups of
people are themselves physical systems. Predictive physics should apply to societies and empires
the way it applies to anything else. Sociophysics is the field that studies the behavior of social
systems, and econophysics studies economic systems. Statecraft sometimes leverages
cybernetics through metaphysical or cultural symbols rather than purely mathematical ones, but
at the most general level we can approach the question from physics.
Complex Adaptive Systems and the Institutional Superstructure
Any society, whether it is the Roman Empire, the Soviet Union, China, or the United States, is
what the literature calls a complex adaptive system. To function beyond anarchy, a society must
have institutions that enable cooperation past the Dunbar limit, which is the rough cap on the
number of stable social relationships an individual can maintain. Above that limit, you need
institutions to prop things up and stabilize behavioral norms.
These institutions are well described by Luhmann's systems theory and include things like
media, law, technology, entertainment, religion, culture, and even medicine. You can also read
them through Marx as the superstructure of society. Either way, the structure is typically
governed by a meta-narrative that rationalizes the governance regime. Terrorism, climate
change, and pandemics function as recent examples. The pattern of history described by
philosophers like Hegel is cyclical. Societies rise from a crisis, bifurcate into establishment
forces and an alienated insurgency, and through conflict collapse into a new order. The United
States shows this clearly. About every eighty years, after a certain number of generations has
passed, a major war or conflict fundamentally restructures society. The American Revolution,
the Civil War, World War II. The norms for a civilization are set by the victors at these inflection
points and over time become less reflective of the needs and desires of the population.
If you believe theorists like Ray Dalio or Strauss and Howe, this pattern is predictable and
ultimately driven by interest on debts beginning to supersede a society's ability to pay them. The
consequence is that societies become corrupted and closed off as power and wealth concentrate
into fewer hands, with more resources going toward maintaining institutional stability than
toward servicing their own people. At this late stage, an empire has the option of turning inward
in the form of civil war, mass surveillance, and internal conflict, or otherwise turning outward in
the form of imperialist wars of expansion.
AI as a Control Loop on a Complex Society
In late-stage societies, which are characterized by extreme complexity, vast institutional
networks, and dense information flows, there is a growing reliance on AI systems to serve as an
information, surveillance, and control loop. An ouroboros, or in mathematical terms a non-
commutative torus. These systems ingest enormous quantities of social, economic, and
behavioral data and compress it into signals that institutions use to make decisions, maintain
order, and project stability. Central planners, or anyone tasked with leading a society, must
gather data on the population to maintain alignment, whether through votes, price signals, or
surveillance. As a society grows and progresses, complexity (entropy) accrues, and the task gets
harder until you hit a point of diminishing returns where institutions can collapse or leaders can
be overthrown. These are referred to variously as bifurcation points, tipping points, catastrophe
points, self-organized criticality, or emergence.
Maintaining institutional alignment is a difficult task, like balancing on the head of a pin, and it
sits at the intersection of nonlinear systems theory and quantum chaos theory. Game theory and
its quantum adaptations are sometimes used where the interests of capital owners and workers
are reconciled through models reminiscent of quantum gravity, which itself is an attempt to
bring together quantum field theory with general relativity in the quantization of value. Some of
the mathematics carries over from loop quantum gravity, because you are dealing with a kind of
network of people which is structurally similar to a spin-foam network, and information flow
forms an ouroboros-like loop. The collective behavior of groups can also be understood through
social laser theory.
The quantization of value is an enormously hard task, much like quantum gravity. Think of
credit scoring algorithms, predictive policing systems, algorithmic content curation, high-
frequency trading, and eventually systems that influence policy in near real time. The concept of
artificial general superintelligence is the endpoint of this trajectory. A system capable of resolving what Nick Bostrom calls the alignment problem by aligning AI objectives with human
values across an entire civilization.
Ashby's Law and Variety Attenuation
What makes the current moment particularly interesting from a cybernetics perspective is that
we can be quite precise about how this institutional decay operates. The precision comes from a
concept called the law of requisite variety, developed by the cyberneticist W. Ross Ashby. The
principle is straightforward. A controller can only regulate a system if its range of possible
responses is at least as broad as the range of states the controlled system can occupy. Think of it
like a thermostat. If a thermostat can only do two things, turn the heat on or off, it can only
regulate a room with two states. A more complex environment requires a more sophisticated
controller. Within econophysics, there is a related framework that distinguishes "HOT" and "COLD"
regimes for decision-making under greater or lower risk:
As a society progresses, entropy accumulates in institutions, demanding ever greater bandwidth
from individual agents to maintain. This is sometimes modeled with agent-based network
simulations, up to a point of saturation that Joseph Tainter describes in his book The Collapse of
Complex Societies. What we observe in late-stage societies is what we can call variety
attenuation, or social compression. A deliberate narrowing of the range of responses available to
ordinary people, a pruning of the degrees of freedom they can experience while participating in
society. In media, this is sometimes called the Overton window. Every institution, from schools
to courts to political parties to media organizations, functions as a variety reduction mechanism.
The goal, whether consciously designed or emergent, is to compress the political and social
behavior of the population into a range the governing class can manage. Agents facilitate one-
way flows of information between social and economic institutions, enforced by cryptographic
gates that encode secrets and govern monetary and digital transactions.
The socioeconomic status of agents in a complex society can be modeled with computational
complexity theory and is also studied within complexity economics, including the spectral
theory of value, where the ultimate value of a society is understood in terms of the capacity of its
constituents. This theory has been under active development by the World Economic Forum and
the Santa Fe Institute.
Spectral Collapse
We can model late-stage complex societies using a framework borrowed from non-commutative
geometry and spectral theory. Institutions are treated as operators acting on a high-dimensional
state space of societal information. The eigenvalue spectrum of this system encodes the modes
of value creation and coordination available to the society. A healthy adaptive society has a rich,
diverse eigenvalue spectrum. Many independent modes of operation, many pathways for
problem solving, many channels of lateral communication between agents. Local communities
are strong and the complex behaviors a civilization requires can be maintained. A society is open
when it is in a phase of growth and relative prosperity, where social support systems and trust
are prevalent. As AGI-driven centralization increases, this spectrum contracts. Institutions
become nearly commutative under one central control regime. Many distinct eigenvalues
collapse toward degeneracy. The effective rank of the system approaches one. This is what we
call spectral collapse, and it has a concrete interpretation. The society loses the adaptive
diversity it needs to respond to novel perturbations. It becomes brittle and susceptible to
authoritarianism and to the collapse of the political spectrum.
This is not just about cultural or ethnic diversity. Lateral inter-agent information throughput
can actually become restricted beyond a certain point of complexity saturation, where cultural
alienation becomes a limiting bottleneck, especially as a population ages. A society can fail to
respond quickly and adaptively to evolving needs due to a lack of lateral interconnectivity
between agents. This concern is also reflected in a recent RAND study, Sources of Renewed
National Dynamism, which assessed the condition of the United States. This spectral collapse
sets the stage for what dynamical systems theorists call a fold catastrophe in bifurcation theory,
where political polarization reaches extrema and eventually results in a fold of the political
spectrum, with left-wing and right-wing extremism becoming inverted projections of one
another in a horseshoe-fold catastrophe. The result is authoritarian and totalitarian regimes.
We can model institutional dynamism as a state variable governed by an equilibrium condition
that depends on a control parameter representing AGI intensity. At low control levels, there
exist stable high-dynamism equilibria.
As control increases, the system follows a stable branch that yields diminishing returns. Eventually the stable and unstable branches coalesce and annihilate at a critical point. Beyond that critical point, no high-dynamism solution exists. The system abruptly jumps to a low-dynamism equilibrium. Institutional vitality collapses. This is the mathematical signature of what we can call a gentle technological singularity. Not a runaway intelligence explosion or a robotic uprising, but a phase transition in societal organization where the very control apparatus meant to preserve stability becomes the mechanism of collapse. Critical slowing down is the observable precursor to this transition, and it should already be visible in the data. Lengthening legislative cycles, growing regulatory backlogs, increasing volatility in governance metrics, intense political polarization and alienation, delays in family formation, institutional breakdown of norms, and an aging population. Evidence suggests these signals are already present in several advanced economies. We can formalize this using the Von Neumann entropy of the normalized institutional operator. Entropy decreases monotonically with control intensity, indicating systemic contraction into a low-entropy, fragile state.
The Economy as an Orchestra Losing Its Sections
The formal way to measure this in an economy is through spectral analysis, looking at the
eigenvalue structure of an economy's productive system. Think of an economy like an orchestra.
A healthy economy has many sections playing different parts simultaneously, including
manufacturing, agriculture, local services, technology, and finance, each representing an
independent mode of activity. Economists like Mariolis have shown empirically that as
economies financialize, this spectrum collapses. Lateral inter-agent interconnectivity reduces
under social and economic constraints, which taps out social capital in trophic social networks
required to buffer high-complexity tasks. Fewer and fewer independent modes remain active.
The orchestra loses section after section until only one instrument is playing louder and louder,
which is financial extraction. This is the dynamic that can produce totalitarian or authoritarian
movements.
An economy with only one dominant eigenvalue cannot adapt to disturbance because there is
only one lever and it can only do one thing. There are no local community structures to buffer
dominant modes. This is directly measurable in the United States. Total factor productivity
growth, a broad measure of how efficiently an economy is actually producing things, averaged
over 2% per year from 1920 to 1970. Since 2005 it has been under half a percent. That is not a
minor slowdown. That is a system that has largely stopped producing and shifted to extraction.
From the Gold Standard to Surveillance Capitalism
What drives this extraction? Here the monetary history becomes important. Value in an
economy needs to be anchored to something. For most of the 20th century it was anchored to
gold, then to dollar-gold convertibility under Bretton Woods. Then after Nixon decoupled the
dollar from gold in 1971, the anchor shifted to geopolitical arrangement, specifically the
petrodollar agreement with Saudi Arabia in 1973 and 1974, which ensured global demand for
dollars by requiring oil to be priced in them. And then increasingly to something even more
abstract, behavioral surplus. The data generated by your social activity, your attention (which
you may have realized comes from papers like Attention Is All You Need, these groundbreaking
papers in artificial intelligence), and your relationships are refined into predictions about your
future behavior and sold.
This is what Shoshana Zuboff calls surveillance capitalism, and it represents the terminal stage
of a long dematerialization sequence where value has progressively detached from anything
physical. The economist Yanis Varoufakis argues we have now passed beyond capitalism entirely
into what he calls technofeudalism. In classical feudalism, lords owned the land and extracted
tribute from serfs who had no option but to farm it. In technofeudalism, the cloud lords,
including Amazon, Apple, Google, Meta, and Microsoft, own the digital infrastructure through
which all contemporary economic and social life is now compulsively routed, including dating,
and they extract what Varoufakis calls cloud rent from everybody who must use it.
The critical distinction from capitalism is this. Capitalism extracted profit from production.
Technofeudalism extracts rent from access. You do not compete with Amazon. You pay Amazon
to exist on Amazon. That is a feudal relationship, not a capitalist one. So you aren't recognized
as a full person unless you have a digital proof of personhood, which you can only access
through something like a smartphone app or a digital identity system.
These platforms also follow a predictable decay cycle that the writer Cory Doctorow has called
enshittification.
1. Stage one — the platform is generally good to users. It builds the network.
2. Stage two — it begins abusing users to extract value for business customers.
3. Stage three — it extracts value purely for shareholders until the platform dies.
We can see this arc clearly with social media, search engines, and e-commerce. The underlying
dynamic is that once a platform has captured enough of the social or economic infrastructure, it
no longer needs to compete on quality. It can charge rent simply for access to the network it has
enclosed.
The Thermodynamic Ceiling on Control
By Landauer's principle, erasing one bit of information requires a minimum energy expenditure
of kB·T·ln(2). An artificial general superintelligence attempting to maintain societal order must
continuously suppress behavioral variance across billions of agents. The energy cost of this
entropy suppression grows at least linearly with the complexity of the system, and likely
superlinearly as control precision increases.
We can identify a tipping point at approximately 1,000 terawatt hours per year of AI compute (which I've made a case for in my preprint you can find on my main homepage) which echoes internal research at OpenAI that comes to similar conclusions:
At this threshold, the marginal energy cost of additional control exceeds the marginal benefit to
stability. The energy return on investment for AI governance falls below 1:1. Data center
electricity consumption was approximately 460 TWh in 2022 and is projected to exceed 1,000
TWh in 2026. AI compute demand doubled roughly every 3.4 months between 2012 and 2018.
My estimate is that we cross this tipping point in 2026, which is this year.
This aligns with several independent frameworks, including data center energy forecasts,
Strauss-Howe generational cycle theory, and Dalio's analysis of debt and governance cycles.
These are not cherry-picked correlations. They represent convergent evidence from very
different analytical traditions, which is also reflected in the recent RAND study.
Reproduction, Family Formation, and the Social Bandwidth Problem
Here we arrive at one of the most important and underappreciated dynamics in our current
moment, which is what happens when all of this converges simultaneously on the domain of
human reproduction and family formation. The U.S. fertility rate reached 1.599 in 2024, the
lowest in recorded history. South Korea is 0.73 and Taiwan is 0.86. LGBTQ affiliation and self-
reported anxiety about meeting expected gender norms set after World War II have increased
substantially. These are not merely cultural preferences. These are measurements of a system
under extreme stress where the bandwidth required for pair bonding and intimacy is limited by the complexity of maintaining institutions.
The mean age at first marriage in the United States has risen from roughly 22 and 21 for men and women in 1950 to over 30 and 28 today. The sociologist Andrew Cherlin's research shows that working-class marriage rates collapsed most sharply in communities hit hardest by deindustrialization, with the causal mechanism being partly material, namely the inability to afford a shared household, and partly about status and shame around failing to perform the traditional provider role.
This also makes sense because in the 1950s it was easier to form long-term stable relationships,
since the average age of the population was 24 after World War II, which means there was less
pressure on young people to prop up an aging population and the accumulated weight of an
economy that has compounded over several generations. What the cybernetics framework
reveals is that this is not a separate social problem from the economic and political dynamics we
have been discussing. It is the same underlying system expressing itself in the domain of
reproduction. The coordination costs of stable pair bonding, including the material
prerequisites, the social scripts, and the bandwidth required, have increased faster than
available capacity. In fact, the same social bandwidth allocated towards inter-agent interactions required for long term stable pair bonding and stable family formation detract from and imperil institutions in their current state because that same social bandwidth is required to maintain them. Although immigration is posed as a solution to this, the result is also wage suppression and does not address social alienation.
Shannon's channel capacity theorem applies directly here. Every social interaction now carries
an institutional and platform noise load that reduces the fraction of bandwidth available for
genuine human exchange. Everything has been commodified and paywalled, including personal
relationships. The libidinal drives of people are sublimated in service of propping up the
economy and both state and corporate institutions. Agents are presented with an exponential
energy gap to achieving the American dream, a kind of carrot and stick where the more agents
traverse the increasingly complex systems of a society, the more their goals seem out of reach.
The dehumanization of people keeps them running on a treadmill deliberately designed to
extract from them.
While gains in women's rights over their bodies to opt out of family formation has sustained increased interest, the choice for stable family formation in the affirmative case is still out of the question for large segments of the population. While gains in the rights of open LGBTQ identification have been made, the ability to build stable long term heterosexual relationships, or even long term relationships of any kind, are strained heavily by socioeconomic precarity. The framing that this is a "choice" people are making to opt out from can be used as a political tool to obfuscate this fact. Framing this entirely as a "choice" hides the brutal socioeconomic reality and represses worker entitlement required to confront institutional power.
Manufactured Polarization
The political polarization we observe follows the same logic. Researchers modeling political
systems using Wasserstein gradient flow, which is a mathematical tool for tracking how
populations distribute themselves across a landscape of options, find that the American political
population has been pushed into a double-well configuration paired with a K-shaped economy.
Imagine a ball on a hill that has two valleys on either side. Once the ball rolls into one valley, it
stays there, and the mechanisms sustaining each camp actively prevent movement toward the
center. This is not an organic polarization. The manufactured bifurcation serves a precise
control function, routing class grievance and social anxiety into intergroup competition that
does not threaten the productive eigenvalue structure generating the underlying instability. At a
tipping point, this may backreact like a spring and imperil institutional stability and the
leadership of a civilization.
Both political camps, regardless of their stated ideologies, behave as infrared- and ultraviolet-
adjacent attractors, representing high-variety chaotic behavior on one end and overly
centralized hierarchical low-variety behavior on the other (in a stagnant socioeconomic
landscape they are inverted projections like a tail of an ouroboros looking into the mouth, or
the mouth looking towards a tail). Both suppress the genuine local community organizing that
would actually be restorative to adaptive capacity. This can also be reflected in the erratic and
seemingly bipolar behavior of leaders.
AI as the Terminal Upgrade of Variety Reduction
This brings us to artificial intelligence, because AI is not entering this situation as a neutral
productivity tool. From the cybernetics perspective, AI is the terminal upgrade of the variety
reduction apparatus, and maybe the final form of neoliberal capitalism. It implements what
physicists call renormalization group flow on human social data, progressively zooming out
from fine-grained local variety, discarding it at each step, and producing a coarse-grained
uniform output. The physicists Mehta and Schwab established an exact mathematical
equivalence between this renormalization process and how deep neural networks function. The
implication is precise. AI applied to human social data systematically destroys the fine-grained
variety on which collective intelligence depends. The Mehta–Schwab paper showing an RG-deep-net correspondence holds for restricted Boltzmann machines under specific conditions which we can use to build a model.
The energy implications alone are staggering. The International Energy Agency projects data
center electricity consumption reaching nearly 1,000 TWh per year by 2030. Meanwhile, the
human brain achieves its cognitive operations at approximately 20 watts. Conservative
estimates put the brain's energy efficiency at hundreds of thousands of times greater per
operation than current GPU-based AI. We are on a course to devote enormous amounts of
energy and trillions of dollars in spending to an information processing architecture that is
trillions of times less efficient than the one evolution produced. And we are doing so precisely in
the domains where human collective intelligence is most needed. The reason for that, even
though it doesn't seem to make economic sense, is because it's in service of propping up
institutions and propping up institutional leadership more than actually servicing the
population.
Where This Leaves Us
Five independent analytical frameworks, including Strauss-Howe generational theory, Hegel's
dialectical theory, Dalio's world order cycle analysis, RAND corporation research on national
power, and Giovanni Arrighi's systemic cycle theory, along with possibly others like Joseph
Tainter's collapse of complex societies framework, independently converge on the current
decade as a critical inflection point.
Dalio declared in March 2026 that the post-1945 world order has officially collapsed. RAND's
research found that no great power has ever recovered from significant long-term decline after it
begins, and they rated the United States as weak or endangered across all eight of their renewal
requirements. But history does not end at collapse. The sociologist Joseph Tainter, who has
studied the collapse of complex societies extensively, documents that societies which do collapse
achieve renewal on the other side through simplified institutions that are more adaptive and
more energetically sustainable. You can understand this from a physics perspective through
what is called conformal rescaling, which in normal terms just means scaling down in size or
complexity.
The question the cybernetics framework poses is whether the transition can be navigated with
sufficient coherence to preserve what is worth preserving, or whether it proceeds through the
kind of chaotic discontinuity that erases institutional memory entirely along with all the
progress we have made as a country. The formal requirements for a positive outcome are
actually quite specific.
1. Restoration of institutional variety through genuine decentralization, local
ownership, and local sovereignty of communities, including federated governance, local
decision-making, and distributed power.
2. Investment in physical community, meaning direct investment in people rather
than scaling up data centers the size of Manhattan, which some of our leaders would like
to launch into orbit.
3. Serious regulation of AI energy consumption and redirection of that capacity
toward the root causes of the instability rather than its symptoms.
Face-to-face interaction is not a sentimental recommendation but a thermodynamic one.
Human interbrain synchrony is the most energy-efficient distributed cognitive infrastructure
available, and social atomization degrades it. Studies show that performance on teams can be
predicted by interbrain synchrony, which is facilitated in person, and that information
throughput in groups scales faster than adding each individual, which is not a feature found in
agentic AI architectures or in architectures where each person is interfacing directly with an AI
tool instead of one another.
Population Inversion: The Singularity Is Us
The closing thought I want to leave you with is this. There is a concept in quantum optics called
population inversion, which is a condition in a laser where there are more atoms in the excited
state than in the ground state, which is what causes the coherent burst of light. Social theorists
have applied this framework to collective social action. When more people have genuinely
recognized the systematic nature of their situation than remain in unreflective acceptance of it,
the conditions exist for a sudden coherent burst of collective reorganization.
Platform decay, institutional failure, and the widening gap between official narratives or meta-
narratives and lived experience are all functioning as the pumping mechanism that drives the
population toward that inversion point, which we might call the technological singularity.
The singularity we should be paying attention to is not an artificial intelligence waking up. It is
actually us waking up. When our leaders can no longer control us by imposing more restrictions,
more controls, more surveillance loops, and so on. In theory, our collective intelligence scales
faster in groups and could reach a critical tipping point after which scaling up AI architectures
reaches a point of diminishing returns, where there is a risk of institutional collapse or anarchy,
which I believe is the correct interpretation of the technological singularity.
The technological singularity in this interpretation is not a point at which AI scales up
exponentially and just becomes so much smarter than everybody else and makes decisions about
what we are doing as a civilization and decides to destroy everybody. More likely it is that while
our collective intelligence scales exponentially and we as a population begin to wake up, the AI
architectures, which are scaling linearly and reaching a point of diminishing returns and
asymptotic limits, are superseded by us, and we no longer decide to follow institutional rules or
norms. There is a serious possibility that there is a kind of crisis or collective anarchy that occurs
because of this.
So in a sense the technological singularity, which has been framed as the moment when
machines wake up and become sentient, is actually when we wake up.
So the technological singularity is actually us. And so they try to invert the narrative as a form of
inverted projection so that they don't really explain why it is that they're so adamant about
scaling up AI. They have an ulterior motive and this would be the motive. It's actually a
cybernetics mass-surveillance control loop.
So this is an inversion of the typical interpretation of the idea of the technological singularity,
but it seems to be a more accurate characterization.
The Possible Outcome
AI is a mass surveillance cybernetics control loop (an ouroboros or "noncommutative torus"). Like any piece of software, when it is deployed the responsibility for harm it may cause rests in the hands of the operator. My personal opinion is that attributing "agency" to "AI agents" is a categorical error. Corporations or governments may claim an AI has "agency" on its own in order to obscure accountability with plausible deniability, outcrowd human agency (like flooding social media with bots designed to appear to be actual users), or reap benefits of content produced by human users with agency to resell it at a price. The sleight of hand is that this makes it possible for operators to claim that software is "autonomous" to avoid personal accountability when it causes harm, but reap benefits for when it succeeds - that by definition means that these "AI agents" are not actually "autonomous."
This is a problem I considered also at Aurora Flight Sciences working on autonomous aerial vehicles. If a drone is "autonomous" and decides to harm a human, is the drone responsible for the death? Obviously, the responsibility would be with the operator of the drone (and this is also the stance the FAA has taken). There was a famous IBM quote to this effect: "a computer can never be held accountable, so a computer must never make a management position." I would go further and state: "a computer ("AI agent") can never be held accountable, and the operator holds responsibility for its deployment." To me, cynically, the idea that AI has "agency" is largely a marketing ploy designed to assert that AI software deserves the same rights as people, and can be used to fool people into accepting control over their lives by the operator of the agents (in a way similar to the idea that "corporations are people" to shield the actual executives from any responsibility).
There is an actual difference between the way the brain works and the way AI models work, and the difference matters, because it can also be measured empirically (collective intelligence in groups scales faster per unit of energy than AI models which reach diminishing returns, when AI models depend on human users for training input and human users are alienated from one another interfacing with AI tools this can lead to a breakdown effect and diminishing returns, interbrain synchrony between human agents increases CI, etc).
The biggest problem I see is the breakdown of the social contract as AI scales, and since AI depends on structured human data, this can thus also cause the AI models to suffocate. This can be understood through cybernetics theory, complex adaptive systems theory, complexity economics, the spectral theory of value, sociophysics, econophysics, catastrophe theory (also related to bifurcation theory or sometimes "emergence" theory).
Any society to function must balance the interests of workers or citizens and the owners or central planners. This is sometimes called "alignment," and it is difficult like balancing on the head of a pin, or sometimes considered similar to the problem of quantum gravity (quantization of value). Corporations and governments have done this by manufacturing consent, and psychological nudging, as well as price signals, votes, and now mass surveillance as feedback loops. As society progresses, alignment requires more feedback due to the entropy that accrues within institutions (by Luhmann's systems theory) until a point of diminishing returns, after which there is usually a collapse of institutions in the form of anarchy (tipping points/catastrophe points). This pattern, regardless of the cybernetics complex adaptive system infrastructure imposed, leads to this inevitable outcome at predictable intervals (the American Revolution, Civil War, WWI/WWII, etc). This is just a result of physics, and says nothing about the morality of the system - the morality of the system will depend on whether you are an insider benefitting from it or become an alienated outsider. After enough agents become disenchanted, social laser theory predicts these tipping points.
The obvious question to ask is that if the human brain operates at only 20 watts of electricity, and collective human intelligence scales faster per unit of energy than AI systems training on data generated by human agents in groups, and dependency of human agents on AI tools diminishes CI, what benefit is there after a catastrophe point (approached as the end point of a period of diminishing returns as a fold catastrophe) for continued investment in these AI systems over direct investment in people? This is an inversion of the popular "technological singularity" metanarrative, where the idea is that AI will scale to a point that it supersedes human level intelligence and then proceeds to take over humanity. In my view, AI is a technological reflection and final form of late stage neoliberalism, and human workers ("agents") have been enslaved by the cybernetics systems established since the end of WWII. At first, the systems allowed agents to build lives of meaning, building families, owning homes, and increasing their standard of living - that has changed especially since at least 2001. A "technological singularity" or "AGI" is thus the point that the vast majority of people or workers decide that the cybernetics architecture imposed onto them no longer holds legitimacy and institutions collapse. That is not a moral argument for or against deployment of AI, that is a thermodynamic inevitability.