A synthesis of insights from books, reports, and expert dialogues (that we have included in previous OODA Loop analysis or were in our research queue) all point to a central theme: enterprises should strategically approach generative AI and agentic AI systems given their rapid evolution, emergent behaviors, and risks of architectural lock-in.

The strategic insights in this summary grow out of the recent September 2025 OODA Network Member Meeting Discussion: Navigating Agentic AI, Security, and Governance, where members confronted how autonomous, non-deterministic AI systems are reshaping enterprise risk, governance, and security in ways that demand new frameworks and foresight.

Overview: The Nature of AI Evolution

Enterprise IT must prepare for architecture churn and emergent behaviors. The organizations best positioned for advantage will be those that build modular, resilient, and adaptive AI ecosystems while continuously monitoring for disruptive evolutionary leaps.

The recent OODA Network monthly discussion on enterprise AI further distilled a core Enterprise AI issue we have been tracking this year – which is the strategic dilemma that is created by the very nature of advanced AI systems: they are black boxes, evolve at biological speed, and display emergent capabilities at scale:

• Continuous Architecture Churn: The very design of AI systems (trained on larger datasets, with growing parameters, reinforced by ecosystem effects) ensures that platforms will not stabilize for long. Each generation is likely to be supplanted by another with qualitatively new emergent behaviors.
• Threshold Surprises: Like biological evolution, new capabilities appear suddenly at scale thresholds. Agentic AI (autonomous systems with task-pursuit and decision loops) represents such a threshold leap beyond generative systems.
• Lock-In Risk: Just as enterprises once locked into mainframes, ERP systems, or cloud providers, locking into an emergent AI architecture risks being trapped when the next evolutionary wave changes the landscape.

For enterprise IT, this means AI cannot be treated as a stable infrastructure layer but must be managed as an evolving ecosystem with continuous architectural churn.

Generative AI delivers productivity acceleration but introduces risks from opacity, hallucination, and data leakage. Agentic AI, the next wave, will add autonomy and orchestration – multiplying both upside and systemic risk. Unlike ERP or cloud layers, AI architectures will mutate rapidly, raising the risk of premature lock-in to platforms that may soon be obsolete.

Drawing from insights in Genesis (Kissinger, Mundie, Schmidt), generative and agentic AI represent both opportunity and risk:

• Modern AI systems function as black boxes, delivering useful outputs without transparent reasoning, which challenges long-held norms of explanation and reproducibility.
• Their evolutionary speed mirrors biology but operates on digital timescales, allowing AI to absorb knowledge across domains and uncover new conceptual insights at rates the human brain cannot match.
• As models scale, emergent capabilities appear unpredictably at certain thresholds of complexity, producing novel behaviors that even their designers cannot anticipate.
• Taken together, today’s frontier models are both opaque and explosively generative, with threshold effects that make AI feel like a form of evolution unfolding in silicon – according to the authors, akin to a “Cambrian Explosion” of emergent AI structures and behaviors.

When framed by Mustafa Suleyman’s Coming Wave perspective, enterprise IT should assume that AI platforms (first generative, then agentic) are not static technologies but evolving ecosystems that will undergo successive waves of emergence, surprise, and obsolescence.

Mustafa Suleyman frames modern AI as a governance challenge rooted in black-box opacity, scale-driven emergence, and evolutionary speed. He argues that AI systems are largely inscrutable even to their creators, and as they scale from billions to trillions of parameters, they develop unexpected new capabilities that cannot be forecasted from smaller models. This “digital echo of the organic world” drives diversification and acceleration, spreading rapidly across domains in ways that risk overwhelming existing institutions.

Because prediction is unreliable, Suleyman calls for “containment”: a layered mix of technical safeguards, norms, licensing, and regulation to manage misuse and systemic shocks. He positions this response against ongoing academic debates about whether emergent abilities are real or artifacts, urging policymakers and enterprises to assume the worst and prepare for surprise. The bottom line: the black-box nature and sudden leaps of capability make containment a necessity for navigating the coming wave.

When scaled to exponential complexity, AI systems exhibit evolutionary-like speed and diversification of behaviors. This is compared to natural evolution’s ability to generate emergent properties, positioning AI as a digital echo of organic processes where new capabilities unfold once critical thresholds of complexity and power are crossed.

According to Google DeepMind CEO Demis Hassabis, AI systems like DeepMind’s Veo are demonstrating the ability to simulate physics (fluids, lighting, materials) without explicit rules. These “black box” models uncover hidden structures of reality, suggesting a form of intuitive physics similar to human common sense.

For Hassabis, “emergence” is framed as lying on a spectrum. Some phenomena, like cellular automata, can be efficiently simulated by classical computers, while chaotic systems may resist prediction. Still, AI systems keep surprising researchers by efficiently modeling spaces once thought intractable, such as protein folding or weather dynamics. This suggests that black-box AI, through sheer parameter scale and training on vast data, may discover kernels of structure in seemingly chaotic systems – allowing for emergent behavior that feels organic.

Kevin Kelly’s work in Out of Control (1994) and Bootstrapping Complexity (an essay that later fed into the same line of thought) doesn’t use today’s language of “LLMs,” “parameter scaling,” or “emergent behavior” in AI – but the core ideas are present in his framing of self-organizing systems, black-box unpredictability, and the organic parallels of technological evolution.

Kevin Kelly’s Out of Control and Bootstrapping Complexity anticipated the black-box nature of AI, the exponential speed of emergent capabilities, and the threshold-based diversification of behavior. He frames this as technology echoing biology – a self-organizing, evolutionary force that grows complexity and intelligence beyond human foresight.

Kelly warns that such systems will resist top-down control. The best strategy is to guide, cultivate, and harness their self-organization rather than try to engineer them rigidly. This is prescient for today’s AI enterprise architecture problem: organizations risk “lock-in” on emergent architectures that may quickly be supplanted, since the evolutionary churn is intrinsic to the system’s nature.

Adaptive Workflows: Just as ecosystems evolve around niches, enterprises will evolve workflows around agents. Over time, human work may mirror ecological adaptation, with “agent niches” embedded in organizations; and Unpredictable Outcomes: Like in biology, small changes in inputs (data, architecture, orchestration) can yield disproportionate, emergent effects (black swan behaviors that are not easily forecastable from first principles).

A recent a16z podcast discussion further reinforced and validated our current analysis, specifically noting:

• AI is a nonlinear, opaque system where new capabilities appear unexpectedly once complexity thresholds are crossed.
• The emergence of specialization and division of labor as AI scales, echoing biological systems.
• Cautioning against linear predictions, noting that exponential dynamics make forecasting specific milestones futile; and
• Acknowledging the risk of lock-in and unpredictability, where organizations and societies may overcommit to architectures that later evolve unpredictably.
• Acceleration Curve: They warn against fixating on dates like “AI 2027” because exponential progress is nonlinear and unpredictable – resonating with the evolutionary metaphor of sudden jumps in capability once thresholds of parameters, compute, or architecture are crossed.
• Parameter Scale → New Capabilities: Just as large parameter counts enable emergent skills (e.g., math reasoning, translation, coding), the a16z discussion highlights how subdividing agents or scaling context windows leads to qualitatively new behavior – long-running persistence, orchestration, task division.
• Platform Shifts as Complexity Emergence: They compare AI to historical platform shifts (PCs, internet, mobile). In each case, once complexity thresholds were crossed, entirely new forms of work emerged. Here, the “black box” agent complexity is doing the same: redefining workflows rather than just accelerating them.

Why This Matters

Enterprises are investing heavily in AI platforms, yet the ground is shifting underfoot. The very dynamics of AI evolution (emergence, speed, and unpredictability) demand new governance, foresight, and architectural strategies. Organizations that assume stability risk-embedding technical debt; those that anticipate churn will be positioned to adapt and thrive.

AI systems are evolving like organic ecosystems – black boxes whose emergent capabilities accelerate as models scale. Enterprises cannot treat Generative AI and Agentic AI as one-off technology waves but must instead prepare for ongoing architecture churn, risk of vendor lock-in, and the imperative of containment.

Emergent Surprise

• New capabilities surface suddenly as models cross parameter thresholds.
• This undermines static risk models and requires enterprises to plan for discontinuous jumps in capability.

Strategic Volatility

• Enterprise AI stacks will be disrupted by successive platform evolutions.
• Current investments may be overtaken within 18–36 months, creating churn risk.

Containment Imperative

• Like Mustafa Suleyman’s call for societal containment, enterprises must build layered safeguards.
• The priority is bounding operational risk while still enabling innovation.

Lock-In Risk

• Single-architecture reliance can entrench systemic vulnerabilities.
• Risks include technical fragility, regulatory exposure, and reputational harm.

Scientific Frontier

• AI could become the ultimate instrument for uncovering hidden informational structures.
• This positions AI as both a research accelerator and a paradigm-shifting scientific tool.

Strategic Leverage

• Mastery of black-box AI architectures impacts national competitiveness, energy security, and healthcare transformation.
• Control of AI architectures equates to geopolitical and economic leverage.

Civilizational Choice

• The trajectory of AGI may determine whether humanity enters an era of radical abundance or faces destabilizing risks.
• This is a civilizational inflection point.

Black Box Reality

• Enterprise AI cannot be fully transparent.
• Opacity is a feature of complexity, not a bug, and must be managed, not eliminated.

Emergence Is Inevitable

• Like ecosystems, new capabilities will arise unexpectedly as models scale.
• Attempts to halt emergence are futile; strategy must focus on adaptation.

Evolutionary Churn

• Enterprises must prepare for rapid obsolescence in AI platforms as architectures evolve.
• Planning cycles must assume constant change.

Governance Challenge

• Oversight should focus on guiding behaviors and managing outcomes.
• Prescriptive control of mechanisms will fail in the face of black-box complexity.

Key Points

Architecture Churn is Inevitable. IT leaders should anticipate re-platforming cycles similar to, but faster than, the cloud era.

Black-Box Nature

Enterprises must work with tools they cannot fully explain, balancing utility with governance.

AI operates as a black box, producing outputs without transparent reasoning. This opacity is not a flaw but a feature of complexity itself, validating early predictions from Kevin Kelly that advanced systems would resist human comprehension. The result is a profound challenge for trust, audit, and compliance.

Emergent Thresholds

The phenomenon of “emergent surprise” undermines static risk models, requiring enterprises to plan for discontinuous leaps in capability rather than gradual progress.

AI systems evolve nonlinearly, with capabilities appearing suddenly as complexity thresholds are crossed. These tipping points echo biological evolution, where variation and selection create new functions unexpectedly.

Generative → Agentic Shift

Generative AI is valuable but risk-prone, demanding careful monitoring and safeguards around data integrity and reliability. The next phase – Agentic AI – brings autonomy, orchestration, and cascading system-of-systems risks.

Enterprises must anticipate this progression: today’s concerns about content reliability will quickly evolve into tomorrow’s concerns about autonomous decision execution and cascading operational consequences.

Architecture Churn & Lock-In

The strategic posture must be modular, multi-model, and open by design to mitigate these risks.

Enterprise AI stacks will not stabilize; they will churn and evolve faster than prior technology transitions, such as cloud. Proprietary ecosystems and vendor lock-in create systemic vulnerabilities, especially if dominant architectures prove unsafe, obsolete, or non-compliant.

Containment Imperative

Crucially, governance must be continuous; annual reviews are insufficient for systems whose properties change unpredictably.

Like Suleyman’s societal containment framing, enterprises must establish layered safeguards to bound operational risk. This requires technical measures such as sandboxing and monitoring, organizational oversight through AI risk boards and red teams, and contractual levers like exit clauses.

Strategic Hedging

The guiding principle is cultivation over control: nurture adaptive ecosystems rather than dictate rigid designs.

Modularity is the most effective hedge against architectural volatility. API-first, multi-cloud, and open-standard strategies preserve agility, ensuring enterprises can pivot as platforms shift. Swarm architectures (distributed networks of agents) may soon supplant centralized models, offering resilience but multiplying governance challenges.

Strategic Foresight

Strategic advantage will accrue to those who anticipate both breakthroughs and bottlenecks, integrating AI into broader futures planning.

Enterprises must treat AI adoption as a foresight discipline, continuously tracking trigger events, emergent architectures, and scenario shifts. Constraints such as energy bottlenecks may slow progress, while the “taste and creativity gap” highlights limits in AI’s ability to generate profound insights.

To access the books and discussion on which this analysis is based, see:

• The September 2025 OODA Network Member Meeting: Navigating Agentic AI, Security, and Governance
• Out of Control – Kevin Kelly
• Genesis – Henry Kissinger, Craig Mundie, Eric Schmidt
• The Coming Wave – Mustafa Suleyman
• A16z AI Agent Future Discussion
• Demis Hassabis (CEO of Google DeepMind): Future of AI, Simulating Reality, Physics and Video Games
• OODA Loop Swarm Architecture Analysis

What Next?

Treat AI as an evolving ecosystem, not a one-time infrastructure investment.

1. Generative AI → Agentic AI: From Tools to Actors

• Generative AI is largely about producing outputs (text, images, code, etc.) in bounded ways. It can be integrated into existing enterprise stacks as a service or co-pilot layer.
• Agentic AI, by contrast, introduces autonomy, orchestration, and decision-making. Instead of just “generating,” these systems act, learn, and coordinate – across workflows, APIs, or even with other agents.
• The shift means moving from applications to ecosystems: agentic AI will require IT architectures that can host, monitor, and constrain agents much like microservices – but with emergent, less predictable behavior.
• Enterprise IT must treat this not as application adoption but as a platform shift, akin to the PC, internet, and cloud revolutions.
  • Generative AI (content creation, copilots) is the first phase: humans stay in the loop, verifying outputs.
  • Agentic AI (autonomous, persistent, orchestrated systems) is the second phase: AI begins to reshape workflows and assume background responsibilities.

2. Continuous Architectural Evolution

• AI platforms will not stabilize the way ERP or cloud platforms eventually did. Each evolutionary leap (e.g., GPT → multimodal → agents → swarm architectures) represents a qualitatively new substrate.
• This means architectural review must be ongoing, not episodic. CIOs should expect that today’s “enterprise AI platform” may be supplanted within 18–36 months.
• Guardrails, interoperability standards, and modular design will be critical, so organizations don’t get locked into an emergent architecture that may later prove unsafe, costly, or strategically limiting.

3. The Lock-in Trap

• Risk: Vendors will attempt to lock enterprises into proprietary agent ecosystems, much like cloud lock-in, but with higher stakes since agents encode workflows, data flows, and decision-making.
• Mitigation: Enterprises should insist on open standards for agent orchestration, model interchangeability, and observability. This prevents being trapped in an architecture that was optimized for one wave of AI but fails in the next.
• Analogy: Think of adopting agentic AI like adopting containerization a decade ago – the architecture mattered as much as the tool, and open frameworks like Kubernetes reduced long-term lock-in risk.
• Lock-in risk is not just technical (APIs, SDKs) but operational: training employees, designing workflows, aligning governance to one architecture.
• Because agentic AI reshapes workflows, switching costs compound: if you design your enterprise around Vendor X’s orchestration model, unwinding it later may be harder than switching ERP in the 2000s.
• Mitigation: Pilot in bounded domains; document workflows in a vendor-agnostic way; maintain a “dual-stack” readiness (test a secondary vendor’s stack in parallel).

4. Black Box + Emergent Diversification = Operational Risk

• AI systems at scale generate capabilities that were not explicitly designed for. Enterprises must assume unexpected emergent behavior will surface in production.
• This necessitates “AI architecture red teaming”: structured review exercises to test for lock-in risks, emergent vulnerabilities, and systemic failures.
• Monitoring must expand beyond uptime and performance to include emergent behavior detection, agent decision audits, and systemic stress-testing.

5. Strategic Guidance for Enterprise IT

• Adopt Layered Modularity: Separate the foundation model layer, the orchestration/agent layer, and the enterprise workflow layer. This enables swapping out components as architectures evolve.
• Plan for Short Half-Lives: Assume AI platforms will turn over faster than traditional IT investments. Budget and plan for churn.
• Build Interpretability Into Procurement: Demand visibility into black-box operations – or the ability to layer third-party interpretability tools on top.
• Develop “AI Exit Strategies”: Just as enterprises today have multi-cloud strategies, they will need multi-model, multi-agent strategies to avoid being trapped in a single evolutionary branch.
• Track Trigger Events: Breakthroughs in agent coordination, failures of agent security, or the emergence of swarm-level architectures could rapidly reset the market.
• Architect for Churn, Not Permanence: The very nature of AI (black-box systems with emergent behaviors and rapid parameter scaling) guarantees that architectures will not be static.
• Today’s platforms (open vs. closed-source, model providers, orchestration layers) may be supplanted by new paradigms: specialized sub-agents, vertical platforms, or hybrid orchestration layers.
• Strategic implication: Avoid hard operational lock-in. Architect modularly with API-driven and data-portable designs. Invest in integration flexibility, not single-vendor depth.
• Risk Management through Layered Abstraction: Think in layers of control:
  • Model Layer (foundation models, fine-tunes)
  • Agentic Orchestration Layer (task division, context management, verification)
  • Enterprise Workflow Layer (business processes, compliance, governance)
  • Enterprises should own and harden the top two layers (or at least the orchestration logic) while treating the foundation model layer as interchangeable. This mitigates exposure to the churn at the base.

6. Adopt a Portfolio Mindset

• Treat AI adoption like R&D portfolio management, not IT procurement.
• Run multiple parallel experiments: one generative (content automation), one agentic (background task orchestration), one vertical (domain-specific agent).
• Use results to drive iterative governance frameworks: a rolling, adaptive “AI architecture review board” rather than a one-off platform choice.

7. Prepare for Emergent Disruption

• Emergent properties mean capabilities will appear unpredictably once complexity thresholds are crossed (e.g., reasoning, coding, multi-agent orchestration).
• Enterprises should invest in rapid evaluation functions (tech scouting, red-teaming, sandboxes) that can quickly test and absorb new capabilities.
• This is analogous to cyber “threat hunting” but applied to platform opportunities and risks.

Further considerations

Strategic Outlook

• Strategic Implication: Enterprises and governments must expect AI systems to evolve unpredictably as scale increases, much like living ecosystems.
• Expect Emergence: Plan for “unknown unknowns” in AI capability roadmaps.
• Opportunity Space: Harnessing evolutionary speed could unlock unprecedented scientific discovery and creativity, if guided by governance and interpretability frameworks.
• Risk Dimension: Black-box opacity means emergent behaviors may be beneficial (e.g., protein folding) or destabilizing (unexpected strategies, misuse).

Foresight and Scenario Planning

• Embed Foresight into IT Governance: Treat AI evolution as a foresight challenge – track trigger events (e.g., sudden model breakthroughs, regulatory shifts).
• Scenario Plan Continuously: Use foresight frameworks to anticipate breakthroughs, consolidations, and paradigm shifts.
• For Strategists: Anticipate phase transitions in capability; monitor for threshold jumps rather than linear progression.

Simulation and Red-Teaming

• Pursue Simulation as Strategy: Modeling complex systems (cells, weather, ecosystems) offers immediate scientific and commercial payoffs.
• Institutionalize Red-Teaming: Run simulations on agentic AI deployments to anticipate cascading risks.
• For Security Leaders: Treat AI like a complex adaptive adversary – build defenses resilient to unpredictable emergent behavior.

Modularity and Flexibility

• Design for Modularity: Architect AI stacks for substitutability and vendor flexibility.
• Adopt Multi-Model Strategies: Balance closed and open-source models to diversify capability and reduce lock-in.
• Architect for Flexibility: Design enterprise AI stacks with modularity and exit options to avoid entrapment in obsolete architectures.
• For CIOs & Enterprise Architects: Build multi-platform resilience – avoid overcommitting to a single emergent AI ecosystem.

Exit Strategies and Resilience

• Invest in Exit Options: Ensure contracts, architectures, and data strategies allow for rapid pivots between platforms.
• Develop Exit Ramps: Build pathways out of vendor ecosystems to avoid architectural entrapment.
• Implement Resilience-First Principles: Use circuit breakers, sandboxing, and human-in-the-loop controls to bound risk.
• Establish Annual Re-Baselining: Reassess AI platforms each year for risk, capability, and compliance.

Governance and Oversight

• Governance by Design: Create adaptive oversight systems that account for emergent behaviors and evolving regulations.
• Shift Oversight Models: Move from control-based governance to adaptive frameworks that monitor behavior and outcomes.
• For Policymakers: Create adaptive regulatory frameworks that evolve with system capabilities, not fixed technical rules.

Ecosystem Thinking

• Map the Ecosystem: Track how emergent architectures (e.g., swarm or mesh models) may supplant today’s hub-and-spoke designs.
• Leverage Ecosystem Thinking: Treat AI development like managing an ecosystem – enable diversity, resilience, and feedback loops.

Research and Frontier Development

• Invest in Hybrid Approaches: Combine large models with evolutionary search or reasoning frameworks to unlock emergent novelty.
• Prioritize Transparent Pathways: Advance research in explainability and interpretability to move beyond black-box opacity.
• Design Lighthouse Tests for AGI: Establish benchmarks for creativity, hypothesis generation, and world modeling beyond task-based evaluations.

Leadership and Education

• Educate Leadership: Ensure boards and executives understand AI as a dynamic system, not a static product line.
• For Enterprise Leaders: Build literacy in emergent AI behaviors to guide investment and governance.

Additional OODA Loop Resources

Reducing Agentic AI Risk in the Enterprise: A Playbook for Corporate Leaders

As artificial intelligence continues to transform all aspects of our economy, business leaders from every industry have been seeking ways to improve competitiveness and take advantage of the business value of these new technologies. Some of the most impactful variants of AI are those known as Agentic AI. This approach, which involves the use of software that can act independently, has been around for over two decades, but has recently been empowered by the rise of Generative AI, Large Language Models, and Natural Language Processing. All indications are that we are going to face a Cambrian Explosion in Agentic AI solutions for enterprises.