OpenAI, a San Francisco-based research and deployment company, released GPT-3 in June of 2020  – and the results were instantly compelling: Natural language processing (NLP) with a seeming mastery of language that generated sensible sentences and was able to converse with humans via chatbots.  By 2021, the MIT Technology Review was proclaiming OpenAI’s GPT-3 a top 10 breakthrough technology and “a big step toward AI that can understand and interact with the human world.”

Initially, access to GPT-3 was a selective process complete with a waiting list.  It has since commercialized in collaboration with Microsoft.   In response, EleutherAI – a self-described “grassroots collective of researchers working to open-source AI research”  launched GPT-J in July 2020 as a quest to replicate the OpenAI GPT collection of models.  The goal is to “break the OpenAI-Microsoft monopoly” through broadening availability and the collective intelligence of open-source development of a competing class of GPT models.

GPT is an acronym for “generative pre-trained transformer.” The first paper on the” GPT of a language model was written by Alec Radford and colleagues, and published in a preprint on OpenAI’s website on June 11, 2018.  It showed how a generative model of language is able to acquire world knowledge and process long-range dependencies by pre-training on a diverse corpus with long stretches of contiguous text. (1)

What Next?

• Open-Source collaboration provides low barriers to entry for your organization, providing an on-ramp for your natural language processing efforts.  It is also provides opportunities for collective problem solving and collaboration with the NLP innovation community.
• The Ongoing Cybersecurity Crisis – Security Will Matter:  Do not make any assumptions about the security of your source data and/or the storage of your uniquely trained large language models (LLM) and GPT-X innovations sitting on a cloud-based server in a PaaS model.  Put some thought into the strategic risk faced by your data storage architecture.   An AI NLP middleware ecosystem (based on the same Web 2.0 technologies that are being deployed without proper consideration for cybersecurity best practices) would replicate some of the cybersecurity problems we are seeing in the DeFi and crypto marketplaces and exchanges.  What are the probability and impact of the unauthorized retrieval, damage, public release, ransoming, and/or loss of proprietary AI LLMs, source data repositories, and/or the uniquely designed parameter datasets and models? To say nothing of a breach of the outcomes:  proprietary, AI-based predictive datasets and models?

OODA is here to help.  Please contact us here.

Further Resources

OODA CTO Bob Gourley recently provided a discussion of the potential impacts and use cases of improved NLP.  See   What Leaders Need to Know About the State of Natural Language Processing:  This post seeks to illuminate major developments in computer language understanding in a way that can help enterprise and government leaders better prepare to take action on these incredible new capabilities. Major improvements in the ability of computers to understand what humans write say and search are being fielded. These improvements are significant and will end up changing just about every industry in the world. But at this point, they are getting little notice outside a narrow segment of experts.

GPT-J operates in the context of a larger commercial ecosystem of GPT products and platforms.  See The Current AI Innovation Hype Cycle: Large Language Models, OpenAI’s GPT-3 and DeepMind’s RETRO:  For better or for worse, Large Language Models (LLMs) – used for natural language processing by commercial AI Platform-as-a-Service (PaaS) subscription offerings – have become one of the first “big data” applied technologies to become a crossover hit in the AI marketplace. OpenAI, a San Francisco-based research and deployment company, released GPT-3 in June of 2020 – and the results were instantly compelling. By 2021, the MIT Technology Review was proclaiming OpenAI’s GPT-3 a top 10 breakthrough technology and “a big step toward AI that can understand and interact with the human world.”

Further GPT-J Resources

Can’t Access GPT-3? Here’s GPT-J — Its Open-Source Cousin

GPT-J-6B: An Introduction to the Largest Open Source GPT Model | Forefront

EleutherAI

Frequently Asked Questions | EleutherAI

For more reading on computer understanding of language

Efficient Estimation of Word Representations in Vector Space: This 2013 work enabled unsupervised training of models in a manner that enabled the mapping of semantic relationships between words into a multi-dimensional meaning space where the distance between concepts can be measured. This allows information retrieval to take into account the relationship between different words and sentences.

Attention Is All You Need: This 2017 paper captures methods that enable transformer networks. Transformer networks incorporate the advantages of recurrent neural networks in ways that account for the sequential nature of language.

Open Sourcing BERT: State of the Art Pre-Training for Natural Language Processing: BERT introduced the idea of unsupervised training for language models, which is similar to how our brains work.

SQuAD: 100000 questions for machine comprehension of text: The Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset consisting of 100,000+ questions posed by crowd workers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. It can be used as a benchmark for machine comprehension of text and natural language processing capabilities.

Related Reading:

Explore OODA Research and Analysis

Use OODA Loop to improve your decision-making in any competitive endeavor. Explore OODA Loop

Decision Intelligence

The greatest determinant of your success will be the quality of your decisions. We examine frameworks for understanding and reducing risk while enabling opportunities. Topics include Black Swans, Gray Rhinos, Foresight, Strategy, Stratigames, Business Intelligence, and Intelligent Enterprises. Leadership in the modern age is also a key topic in this domain. Explore Decision Intelligence

Disruptive/Exponential Technology

We track the rapidly changing world of technology with a focus on what leaders need to know to improve decision-making. The future of tech is being created now and we provide insights that enable optimized action based on the future of tech. We provide deep insights into Artificial Intelligence, Machine Learning, Cloud Computing, Quantum Computing, Security Technology, Space Technology. Explore Disruptive/Exponential Tech

Security and Resiliency

Security and resiliency topics include geopolitical and cyber risk, cyber conflict, cyber diplomacy, cybersecurity, nation-state conflict, non-nation state conflict, global health, international crime, supply chain, and terrorism. Explore Security and Resiliency

Community

The OODA community includes a broad group of decision-makers, analysts, entrepreneurs, government leaders, and tech creators. Interact with and learn from your peers via online monthly meetings, OODA Salons, the OODAcast, in-person conferences and an online forum. For the most sensitive discussions interact with executive leaders via a closed Wickr channel. The community also has access to a member-only video library. Explore The OODA Community