The release and success of ChatGPT have thrown policymakers into a frenzy of AI regulatory activity. Meanwhile, as with privacy and crypto regulation, the EU is setting the global standard with the recent passage of the EU Artificial Intelligence Act. Over the course of 2021 – as we reviewed a variety of research efforts concerned with AI Risk, AI Ethics, and Trustworthy AI – we made the following recommendations at the enterprise level:
In a July 2021 policy brief “AI Accidents: An Emerging Threat – What Could Happen and What to Do,” the Center for Security and Emerging Technology) (CSET) makes a noteworthy contribution to current efforts by governmental entities, industry, AI think tanks, and academia to “name and frame” the critical issues surrounding AI risk probability and impact.
“AI Accidents” framework from the Georgetown University CSET
“For the current enterprise, as we pointed out as early as 2019 in Securing AI – Four Areas to Focus on Right Now, the fact still remains that “having a robust AI security strategy is a precursor that positions the enterprise to address these critical AI issues.” In addition, enterprises that have adopted and deployed AI systems also need to commit to the systematic logging and analysis of AI-related accidents and incidents.
Look no further than this CSET policy brief as a blueprint for such AI accident notification, reporting, and analysis efforts. To foster open collaboration within your organization, the opportunity also exists to build your reliability and safety engineering activities in collaboration with the open innovation and crowdsourcing database (with a very compelling open-source taxonomy architecture) on which CSET authors Zachary Arnold and Helen Toner have based their AI accidents framework and policy recommendations: The Artificial Intelligence Incident Database (AIID) – a project housed at the Partnership on AI.”
Recently, Senator Chuck Schumer announced plans for AI regulatory legislation in the U.S.:
“Schumer said that during the first half of the year, he and his team have been holding discussions with more than 100 AI developers, executives, scientists, workforce experts, and others to develop their legislative framework.
The “SAFE Innovation for AI” framework has five central pillars:
- Security: Shore up national security by examining AI threats from foreign adversaries or rogue groups and ensure economic security for workers, especially low-skilled, low-income workers.
- Accountability: Support the creation of “responsible” systems to address issues like misinformation and bias and support creators by addressing copyright concerns and protecting intellectual property.
- Foundations: Require that AI systems align with democratic values at their core, protect elections, promote AI’s societal benefits while avoiding potential harms, and stop the Chinese Communist Party from writing the rules of the road for AI.
- Explain: Companies should share how an AI system arrived at a particular answer in simple and understandable terms so users can better understand why the system produced a particular answer and where it came from.
- Innovation: Support U.S. innovation in AI technologies that focuses on unlocking the potential of AI and maintaining U.S. leadership in the technology.” (1)
Not only in this framework proposed by Senator Schumer but in the general regulatory discussions that have ensued since the open letter calling for a pause on the training of AI systems more powerful than GPT-4 and OpenAI CEO Sam Altman testifying at the U.S. on “Oversight of A.I.: Rules for Artificial Intelligence, we have been surprised to see that the Review Board Model (like the NTSB or the recently launched Cyber Review Board) has not gained much traction (or been mentioned very often at all in the last few months).
As a result, we went back to the AI Incident Database. It has grown and evolved since its launch in 2021 and has the potential to become a private sector, “bottom-up” phenomenon with a platform designed for growth at speed and scale. It has the potential to evolve into a standalone, self-governing AI Review Board. The World Wide Web Consortium (W3C) comes to mind over the often-mentioned NTSB model. The CISA Cybersecurity Alerts and Advisories model probably offers some structural lessons as well.
And why not? The promise of the AI Incident Database is that the learnings from the AI accidents entered into the database can then become standards and guidelines over time (i.e. the aggregation of the AI accidents then becomes a template for mitigation standards and guidelines – possibly even regulations and laws). As the update below on the AI Incident Database illustrates, AI accidents are already occurring and have a mechanism and a taxonomy for reporting.
AI innovation is occurring at exponential speed. There are no good arguments against this the use growth of this platform by the public and private sectors in parallel with governmental efforts to figure out their role in a regulatory framework for Ai. Historically, that process will move at a snail’s pace – and this emerging technology is simply moving too fast to put all our eggs in the governmental regulatory basket. The AI Incident Database has all the design qualities and the ethos of the best elements of the open-source movement – making it the most viable collective intelligence risk mitigation effort relative to the success or failure of governmental policymaking efforts.
AI Incident Roundup – April ‘23
Since our previous analysis in 2021, the AIID has launched a monthly incident newsletter, of which April 2023 is the most recent publicly available version – and representative of the issues the database is currently tackling:
🗄 Trending in the AIID
In April, there were several news stories centering around false accusations and impersonation. Whereas AI-powered voice synthesis systems were being used as tools by humans to impersonate loved ones for cash, break into a government system, make false swat calls, and mimic musicians, ChatGPT generated false stories of an Australian Mayor and a professor committing crimes.
🗞️ New Incidents
LLMs & Chatbots
- #506: ChatGPT Allegedly Produced False Accusation of Sexual Harassment
- LLM, false accusation
- #507: ChatGPT Erroneously Alleged Mayor Served Prison Time for Bribery
- LLM, false accusation
- #513: ChatGPT Banned by Italian Authority Due to OpenAI’s Lack of Legal Basis for Data Collection and Age Verification
- LLM, children
- #514: Turnitin’s ChatGPT-Detection Tool Falsely Flagged Student Essays as AI-Generated
- LLM detection
- #516: ChatGPT Reportedly Exposed Users’ Private Data Reportedly Due to Bug
- LLM, false privacy
Voice Synthesis
- #509: Scammers Deepfaked Videos of Victims’ Loved Ones Asking for Funds over Facebook in Vietnam
- deepfake, scam, voice recognition, voice synthesis
- #523: Australian Journalist Able to Access Centrelink Account Using AI Audio of Own Voice
- privacy, voice recognition, voice synthesis
- #524: AI Voices Abused by Telegram User to Make Swat Calls as Paid Service
- voice synthesis, abuse
- #526: Novel Deepfake Song Pulled from Music Streaming Services After Allegedly Violating Artist’s Rights
- deepfake, voice synthesis
Computer Vision
- #517: Man Arrested For Sock Theft by False Facial Match Despite Alibi
- false accusation, facial recognition, computer vision
- #518: New York Detective Misused Woody Harrelson’s Face to Perform Face Recognition Search
- false accusation, facial recognition, computer vision
- #520: Amazon Fresh Cameras Failed to Register Purchased Items
- computer vision
- #521: Images Captured by iRobot’s Roomba Containing Device Users Posted on Private Online Groups
- computer vision, privacy
Bias & Discrimination
- #522: Facebook Political Ad Delivery Algorithms Inferred Users’ Political Alignment, Inhibiting Political Campaigns’ Reach
- bias, privacy
- #527: Tech Companies Reportedly Paid Gig Workers Differentially for the Same Amount of Work
- discrimination, pricing algorithm
- #529: Stable Diffusion Exhibited Biases for Prompts Featuring Professions
- bias, image generation
- #531: AI-Assisted Body Scanners Reportedly Subjected Transgender Travelers to Invasive Body Searches
- abuse, privacy, discrimination
Other
- #519: Starship Delivery Robot Ran into Problems Traversing Campus Terrains
- #525: Tesla Vehicle Running on Self-Driving Mode Crashes on City Streets
- #528: Amazon Algorithmic Pricing Allegedly Hiked up Price of Reference Book to Millions
- #530: Telegram Channels Allowed Users to Make Non-Consensual Deepfake Porn as Paid Service
👇 Diving Deeper
- Explore clusters of similar incidents in Spatial Visualization
- Check out Table View for a complete view of all incidents
- Learn about alleged developers, deployers, and harmed parties in Entities Page
Welcome to the AI Incident Database – AIID (2023)
The AI Incident Database (AIID) is dedicated to indexing the collective history of harms or near harms realized in the real world by the deployment of artificial intelligence systems. Like similar databases in aviation and computer security, the AI Incident Database aims to learn from experience so we can prevent or mitigate bad outcomes.
You are invited to submit incident reports, whereupon submissions will be indexed and made discoverable to the world. Artificial intelligence will only be a benefit to people and society if we collectively record and learn from its failings.
Why “AI Incidents”?
Intelligent systems are currently prone to unforeseen and often dangerous failures when they are deployed to the real world. Much like the transportation sector before it (e.g., FAA and FARS) and more recently computer systems, intelligent systems require a repository of problems experienced in the real world so that future researchers and developers may mitigate or avoid repeated bad outcomes.
What is an Incident?
The initial set of more than 1,000 incident reports has been intentionally broad in nature. Current examples include,
- An autonomous car kills a pedestrian
- A trading algorithm causes a market “flash crash” where billions of dollars transfer between parties
- A facial recognition system causes an innocent person to be arrested
You are invited to explore the incidents collected to date, view the complete listing, and submit additional incident reports.
Defining an “AI Incident”
The commercial air travel industry owes much of its increasing safety to systematically analyzing and archiving past accidents and incidents within a shared database. In aviation, an accident is a case where substantial damage or loss of life occurs. Incidents are cases where the risk of an accident substantially increases. For example, when a small fire is quickly extinguished in a cockpit it is an “incident” but if the fire burns crew members in the course of being extinguished it is an “accident.” The FAA aviation database indexes flight log data and subsequent expert investigations into comprehensive examinations of both technological and human factors. In part due to this continual self-examination, air travel is one of the safest forms of travel. Decades of iterative improvements to safety systems and training have decreased fatalities 81 fold since 1970 when normalized for passenger miles.
Where the aviation industry has clear definitions, computer scientists and philosophers have long debated foundational definitions of artificial intelligence. In the absence of clear lines differentiating algorithms, intelligence, and the harms they may directly or indirectly cause, this database adopts adaptive criteria for ingesting “incidents” where reports are accepted or rejected on the basis of a growing rule set defined in the Editor’s Guide.
Current and Future Users
The database is a constantly evolving data product and collection of applications.
- Current Users include system architects, industrial product developers, public relations managers, researchers, and public policy researchers. These users are invited to use the Discover application to proactively discover how recently deployed intelligent systems have produced unexpected outcomes in the real world. In so doing, they may avoid making similar mistakes in their development.
- Future Uses will evolve through the code contributions of the open-source community, including additional database summaries and taxonomies.
When Should You Report an Incident?
When in doubt about whether an event qualifies as an incident, please submit it! This project is intended to converge on a shared definition of “AI Incident” through exploration of the candidate incidents submitted by the broader community.
Board of Directors
The incident database is managed in a participatory manner by persons and organizations contributing code, research, and broader impacts. If you would like to participate in the governance of the project, please contact us and include your intended contribution to the AI Incident Database.
Voting Members
Helen Toner: Helen Toner is Director of Strategy at Georgetown’s Center for Security and Emerging Technology (CSET). She previously worked as a Senior Research Analyst at Open Philanthropy, where she advised policymakers and grantmakers on AI policy and strategy. Between working at Open Philanthropy and joining CSET, Helen lived in Beijing, studying the Chinese AI ecosystem as a Research Affiliate of Oxford University’s Center for the Governance of AI. Helen has written for Foreign Affairs and other outlets on the national security implications of AI and machine learning for China and the United States, as well as testifying before the U.S.-China Economic and Security Review Commission. She is a member of the board of directors for OpenAI. Helen holds an MA in Security Studies from Georgetown, as well as a BSc in Chemical Engineering and a Diploma in Languages from the University of Melbourne.
Contributions: AI incident research and oversight of the CSET taxonomy.Patrick Hall: Patrick is principal scientist at bnh.ai, a D.C.-based law firm specializing in AI and data analytics. Patrick also serves as visiting faculty at the George Washington University School of Business. Before co-founding bnh.ai, Patrick led responsible AI efforts at the machine learning software firm H2O.ai, where his work resulted in one of the world’s first commercial solutions for explainable and fair machine learning. Among other academic and technology media writing, Patrick is the primary author of popular e-books on explainable and responsible machine learning. Patrick studied computational chemistry at the University of Illinois before graduating from the Institute for Advanced Analytics at North Carolina State University.
Contributions: Patrick is the leading contributor of incident reports to the AI Incident Database Project.Sean McGregor: Sean McGregor founded the AI Incident Database project and recently left a position as machine learning architect at the neural accelerator startup Syntiant so he could focus on the assurance of intelligent systems full time. Dr. McGregor’s work spans neural accelerators for energy efficient inference, deep learning for speech and heliophysics, and reinforcement learning for wildfire suppression policy. Outside his paid work, Sean organized a series of workshops at major academic AI conferences on the topic of “AI for Good” and is currently developing an incentives-based approach to making AI safer through audits and insurance. Contributions: Sean volunteers as a project maintainer and editor of the AI Incident Database (AIID) project.
Non-Voting Members
- Neama Dadkhahnikoo: Neama Dadkhahnikoo is an expert in artificial intelligence and entrepreneurship, with over 15 years of experience in technology development at startups, non profits, and large companies. He currently serves as a Product Manager for Vertex AI, Google Cloud’s unified platform for end-to-end machine learning. Previously, Mr. Dadkhahnikoo was the Director of AI and Data Operations at the XPRIZE Foundation, CTO of CaregiversDirect (AI startup for home care), co-founder and COO of Textpert (AI startup for mental health), and a startup consultant. He started his career as a Software Developer for The Boeing Company. Mr. Dadkhahnikoo holds an MBA from UCLA Anderson; an MS in Project Management from Stevens Institute of Technology; and a BA in Applied Mathematics and Computer Science, with a minor in Physics, from UC Berkeley.
Contributions: Neama serves as Executive Director to the Responsible AI Collaborative outside his role as product manager at Google. - Kit Harris: Kit leads on grant investigations and researches promising fields at Longview Philanthropy. He also writes about Longview recommendations and research for Longview client philanthropists. Prior to focusing on high-impact philanthropic work, Kit worked as a credit derivatives trader with J.P. Morgan. During that time, he donated the majority of his income to high-impact charities. Kit holds a first-class degree in mathematics from the University of Oxford.
Contributions: Kit serves as a board observer, provides strategic advice, and is RAIC’s point of contact at Longview.
What Next?
Case Study: The Timeline of the Creation of the Cyber Review Board
All hope is not lost on the ability of the U.S. Government to act quickly. Following is a series of posts that capture the workshop in the Spring of 2021 (and the publication generated b the workshop released in November of 2021) that called for the creation of a “Cyber NTSB” – which was then launched in February 2022.
“Over four months in the spring of 2021, over 70 experts participated in a (virtual) workshop on the concept of creating a “Cyber NTSB”. The workshop was funded by the National Science Foundation with additional support from the Hewlett Foundation, and organized by Harvard’s Belfer Center with support from Northeastern University’s Global Resilience Institute.
The first call for the creation of a Cyber NTSB was in 1991. Since that time, many practitioners and policymakers have invoked the analogy, but little has been done to develop the concept. This workshop was carried out with the goal of moving the concept forward.”
We have a choice: A timeline of 1991 through 2022 (30 years) or March 2021 through February 2021 (less than a year) for the creation of an AI Review Board.
2021
Cybersecurity and Cyber Incidents: Innovation and Design Lessons from Aviation Safety Models and a Call for a “Cyber NTSB”
2022
Cyber Safety Review Board Launched by DHS
Cyber Safety Review Board (CSRB)
The Cyber Safety Review Board was established pursuant to President Biden’s Executive Order (EO) 14028 on ‘Improving the Nation’s Cybersecurity‘. The Board serves a deliberate function to review major cyber events and make concrete recommendations that would drive improvements within the private and public sectors. The Board’s construction is a unique and valuable collaboration of government and private sector members and provides a direct path to the Secretary of Homeland Security and the President to ensure the recommendations are addressed and implemented, as appropriate. As a uniquely constituted advisory body, the Board will focus on learning lessons and sharing them with those that need them to enable advances in national cybersecurity. (1)
In July 2022, the CSRB released its first report. See https://www.dhs.gov/news/2022/07/14/cyber-safety-review-board-releases-report-its-review-log4j-vulnerabilities-and
For a pdf of the report: https://www.cisa.gov/sites/default/files/publications/CSRB-Report-on-Log4-July-11-2022_508.pdf
Further Resources
For next steps in ensuring your business is approaching AI with risk mitigation in mind.Artificial Intelligence for Business Advantage.
Looking for a primer on what executives need to know about real AI and ML? See A Decision-Maker’s Guide to Artificial Intelligence.
Securing AI – Four Areas to Focus on Right Now
When Artificial Intelligence Goes Wrong
Improving Mission Impact with a Culture That Drives Adoption
The Cybersecurity Implications of ChatGPT and Enabling Secure Enterprise Use of Large Language Models
The OODA Network on the “Real Danger” of AI Innovation at Exponential Speed and Scale – and not Adequately Addressing AI Governance
March 2023 OODA Network Member Meeting Tackled Strategy (Mis)Informing Regulation, Systemic Failure, and the Emergence of New Risks
NIST Makes Available the Voluntary Artificial Intelligence Risk Management Framework (AI RMF 1.0) and the AI RMF Playbook
AI/ML-enabled Systems for Strategy and Judgment and the Future of Human-Computer-Data Interaction Design
