Rochester Institute of Technology experts have created a new tool that tests artificial intelligence (AI) to see how much it really knows about cybersecurity. And the AI will be graded. The tool, called CTIBench, is a suite of benchmarking tasks and datasets used to assess large language models (LLMs) in Cyber Threat Intelligence (CTI). CTI is a crucial security process that enables security teams to proactively defend against evolving cyber threats. The evaluation tool comes at a time when AI assistants claim to have security knowledge and companies are developing cybersecurity-specific LLMs. For example, Microsoft Copilot has an integrated security platform. Until now, there has been no way to tell if an LLM has the capability to work as a security assistant. “Is the LLM reliable and trustworthy?” asked Nidhi Rastogi, assistant professor in RIT’s Department of Software Engineering. “Can I ask it a question and expect a good answer? Will it hallucinate?” CTIBench is the first and most comprehensive benchmark in the Cyber Threat Intelligence space. The tool is already being used by Google, Cisco, and Trend Micro. “We should embrace using AI, but there should always be a human in the loop,” said Rastogi. “That’s why we are creating benchmarks—to see what these models are good at and what their capabilities are. We’re not blindly following AI but smartly integrating it into our lives”.” In her AI4Sec Research Lab, Rastogi is studying at the crossroads of cybersecurity and AI. She developed CTIBench along with computing and information sciences Ph.D. students Md Tanvirul Alam, Dipkamal Bhusal, and Le Nguyen.

Full report : Researchers develop cybersecurity test called CTIBench which evaluates large language models used in Cyber Threat Intelligence applications.