This post examines AI and Human diplomacy and building trust, cooperation, deception, and strategic reasoning into natural language AI systems

In 2022, Meta AI revealed CICERO, “an AI system that beat 90% of human players at Diplomacy. Unlike Chess, Go, etc., Diplomacy requires multiplayer collaboration – and backstabbing!” Meta AI researchers provide an overview of their breakthrough in natural language processing (NLP) and building trust, cooperation, deception, and strategic reasoning into an NLP system.

Building Trust, Cooperation, Deception, and Strategic Reasoning into Natural Language AI Systems

Diplomacy AI “thread-of-threads” from META AI researchers

AI Pub generated a 9/9 thread based on the release of Meta’s CICERO – an AI system that beat 90% of human players at Diplomacy:   “Meta AI presents CICERO — the first AI to achieve human-level performance in Diplomacy, a strategy game which requires building trust, negotiating and cooperating with multiple players.” Meta researchers p

// Diplomacy AI thread-of-threads //

Yesterday, Meta AI revealed CICERO, an AI system that beat 90% of human players at Diplomacy.

Unlike Chess, Go, etc., Diplomacy requires multiplayer collaboration – and backstabbing!

A CICERO thread-of-threads from Meta AI researchers:

1/9 pic.twitter.com/z66SWRwbAy

— AI Pub (@ai__pub) November 23, 2022

From the thread

1. @polynoamial explains some of the inherent difficulties of AI playing Diplomacy: “Diplomacy is about building trust in an environment that encourages players to not trust anyone… must account for the risk that players might lie, and that players might doubt your honesty.”
2. @adamlerer on why CICERO is a huge breakthrough: “[CICERO] addresses 2 [questions]:
   • How to integrate planning & RL ([reinforcement learning], eg. AlphaZero) with dialogue (eg. GPT-3)
   • How to learn and plan in settings with both competition and cooperation, where agents must understand [human intentions]”
3. For decades, Diplomacy has been viewed as a near-impossible grand challenge in AI due to the complex interplay of strategy, theory of mind, and natural language involved...we present CICERO, the world’s first human-level full-press Diplomacy AI. 
4. Our paper describing a human-level AI for Diplomacy was published in Science.  This is the first human-level AI for a game requiring cooperation through *natural language*. Really proud of what we built and excited to finally share it.
5. @ml_perception with some of the training & implementation details behind CICERO. CICERO passed for human in 40 anonymous online games, and finished in the top 10% of players!

An example of persuasion from our new Diplomacy AI agent Cicero, published in Science today. The bot (Italy) successfully convinces France that it's not planning to attack. pic.twitter.com/kzgr9YxvuM

— Noam Brown (@polynoamial) November 22, 2022

Why Diplomacy?:  A Grand Challenge for AI

From Meta AI:

A game about people rather than pieces:  Diplomacy is a seven-player board game that can be described as a combination of the board game Risk, the card game poker, and the TV show Survivor. Unlike many board games where you just need to be the best at moving pieces around the board, Diplomacy has a cooperative component — the only way to win is by working with other players to capture as much territory as possible. This coordination is achieved through natural language negotiation that occurs before every move in the game.

Why Diplomacy? Advancements in AI gameplay have long served as benchmarks for progress in AI. For decades, researchers have been building simplified variants of Diplomacy gameplay agents without natural language communication capabilities. But no one has ever attempted to build an AI agent that can negotiate with open-ended dialogue. Achieving this grand challenge of AI has simply been beyond the capabilities of what’s existed in AI, until now.

The making of a breakthrough:  Strategic Reasoning + Natural Language:  Within CICERO is a language model integrated with strategic reasoning algorithms that control the generation of dialogue. Our advancements in strategic reasoning and natural language processing together enable effective cooperation with humans — by giving CICERO the ability to understand what other players are trying to achieve, negotiate a plan, suggest shared goals and communicate with strategic intent.

Human-level play in the game of Diplomacy by combining language models with strategic reasoning

From the Science article published by the Meta AI research team:

AI masters Diplomacy:  The game Diplomacy has been a major challenge for artificial intelligence (AI). Unlike other competitive games that AI has recently mastered, such as chess, Go, and poker, Diplomacy cannot be solved purely through self-play; it requires the development of an agent to understand other players’ motivations and perspectives and to use natural language to negotiate complex shared plans.

The Meta Fundamental AI Research Diplomacy Team (FAIR) et al. developed an agent that is able to play the full natural language form of the game and demonstrates performance well above the human average in an online Diplomacy league. The present work has far-reaching implications for the development of cooperative AI and language models for communication with people, even when interactions involve a mixture of aligned and competing interests.

Abstract:  Despite much progress in training artificial intelligence (AI) systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge.

We introduce Cicero, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.

What Next?

Playing a part in the metaverse and beyond:  Gaming has long been a playground for pushing the limits of AI. Every iteration takes us deeper into more immersive types of AI, with new ways to interact in the physical and virtual worlds. As we build for the metaverse, we see potential for technology like CICERO’s to power cooperative agents that can help people learn new skills, negotiate on their behalf, create more immersive social and gaming experiences and connect more meaningfully with those around them…by open sourcing CICERO’s code…[which is available at GitHub].

About the Author

Daniel Pereira

Daniel Pereira is research director at OODA. He is a foresight strategist, creative technologist, and an information communication technology (ICT) and digital media researcher with 20+ years of experience directing public/private partnerships and strategic innovation initiatives.