Anthropic’s New AI Model Can Identify More Software Bugs Than Ever. Microsoft Is Struggling To Fix Them Fast Enough.

from the keeping-up-with-the-ai dept

This story was originally published by ProPublica. Republished under a CC BY-NC-ND 3.0 license.

On an afternoon in mid-May, dozens of Microsoft engineers and their managers gathered online and in a conference room at the company’s Redmond, Washington, headquarters to discuss Project Glasswing.

The tech giant was racing to fix weaknesses in its code that a new AI model known as Mythos was uncovering at an unprecedented clip. The AI behemoth Anthropic, which developed Mythos, had given access to select organizations that make software used by regular people, companies and governments across the world. The goal was to find and fix the vulnerabilities before hackers and adversarial governments like China began using similar tools to find and exploit them for espionage and sabotage.

As the group settled in, one engineer asked the question that loomed over the meeting: Did Mythos “live up to the hype that Anthropic claimed it would have had?”

“Yes,” a manager responded, according to a recording of the meeting viewed by ProPublica.

The version being used by Microsoft, Claude Mythos Preview, was surfacing bugs faster than the tech giant could patch them, and engineers, the manager said, were now in “a mad dash” to close the gap.

One slide in that day’s presentation showed that in April alone, Mythos had uncovered 90 “critical” bugs and 141 “important” ones in SharePoint, Microsoft’s widely used collaboration software. In the first half of May it found even more.

“Please, please, please if your org has any April bugs, drive those down,” engineering manager Hans Andersen implored the group. They had roughly two weeks “to find as many things and do as much good as we can with this access.”

May 31, he explained, “is considered the day when the rest of the world will have caught up.”

The engineers on the call poked at that assertion, with one of them summing up the predicament: “So basically you’re saying if it’s released on June 1, then on June 2 the adversaries will have our bugs?”

Yep, one person responded. Yep, another echoed.

Ever since Anthropic kick-started a national conversation about the bug-hunting power of AI in April, when Project Glasswing was made public, national security experts predicted that the U.S. would have a window of opportunity to fix flaws before adversaries would have similar models capable of discovering the same weaknesses. In late June, the international alliance of intelligence agencies known as the Five Eyes — whose members are the U.S., Australia, Canada, New Zealand and the U.K. — warned in an unusual joint statement that in a matter of months, that window would be closing. But the recording of the Microsoft meeting, along with internal documents reviewed by ProPublica, suggest the day of cyber reckoning may already be here.

Given the deluge of flaws Mythos has identified, Microsoft so far has focused on patching those it considers most dangerous, which are classified critical or important, according to the presentation as well as the company’s own public patch updates. The internal records indicate that Microsoft plans to eventually address “moderate”-severity flaws uncovered by Mythos. The documents made no mention of “low”-severity bugs.

The company’s approach reflects the triage system that is typical in the industry. Just as the sickest patients are the first to be treated in the emergency room, vulnerability triage prioritizes issues that are likely to cause the most damage if exploited by hackers.

But that strategy carries its own risk in this AI-powered bug-finding era, in which new tools are unearthing a record-breaking volume of weaknesses in the products we use every day. Mythos, for example, is able to chain together a string of bugs that build on one another, meaning that the low- and moderate-severity vulnerabilities that remain unpatched could create an opening to carry out devastating attacks.

“The problem now is that you can chain four low-level flaws, and that can equal a high severity,” said Vinh Nguyen, a senior technical adviser to Anthropic and a senior fellow for AI at the Council on Foreign Relations who formerly served as chief AI officer and chief data scientist at the National Security Agency. “If you’re Microsoft, the current triage strategy may be underpricing risks.”

In emailed responses to ProPublica’s questions, Microsoft stood by its approach, saying its triaging decisions are based on a number of factors, including exploitability and the impact on customers. The company presentation did not mention chaining, but a spokesperson told ProPublica that the technique “has long been considered as part of vulnerability assessment and risk analysis.”

Asked about the internal presentation and the then-looming May 31 deadline, the spokesperson downplayed its significance, saying that “accelerated targeting and exploitation of new vulnerabilities is not a new phenomenon.” That said, he added, the comments made during the meeting reflect how the company “feels a sense of urgency to help our customers at this time.”

“What was heard on that call and is true today is that security is Microsoft’s most important priority and teams across the company are prioritizing using AI to discover and remediate vulnerabilities as quickly as possible.”

Microsoft declined to answer questions about how many bugs engineers had patched since the presentation.

Anthropic declined to comment.

The internal Microsoft presentation and accompanying slides predicted that the group of staffers working on SharePoint, which is used by governments and businesses worldwide to manage data and documents, “will be busy for months,” first working through the highest-priority critical bugs then tackling the important ones in August. Microsoft says vulnerabilities it categorizes as critical include so-called worms that can crash systems and spread malware as they race across computer networks. Important ones could result in “compromise of the confidentiality, integrity, or availability of user data” as well as the “availability of processing resources.” After those categories were cleared, the group would begin work on roughly 300 “moderate” bugs, according to the presentation.

While the internal documents reviewed by ProPublica do not include updates on the entire breadth of Microsoft’s offerings, they do give a sense of the scale of the problem. One document noted that, since the company started using Mythos earlier this year, it had collectively found hundreds of bugs that Microsoft categorized as either critical or important in popular products such as Microsoft 365, the Teams conferencing platform and the Copilot AI tool. As of mid-May, most of them had yet to be patched.

“They’re not profound and exotic, but they’re real,” Andersen, the engineering manager, said during the meeting. “And a lot of them are exploitable.”

It’s unclear whether hackers have exploited any specific bug identified by Mythos, but some  have tapped AI to automate attacks and appear to be using Mythos-like tech to find and exploit weaknesses.

There have been outward signs of Microsoft’s internal struggle to deal with the growing list of bugs to be patched. Each month, the company publicly releases fixes for its software vulnerabilities in what’s known as “Patch Tuesday.” In June, it released patches for more than 200 bugs, which industry experts then said was an all-time high. But on July 14, the company blew through that record and released patches for more than 600 bugs. Only seven were categorized as low- or moderate-severity, one of which hackers were actively exploiting, according to Dustin Childs, leader of the Zero Day Initiative bug bounty program, which is part of cybersecurity company TrendAI. The rest were important or critical.

“Well folks. Here we are. The bug apocalypse has fully descended upon us,” Childs wrote in a blog post on July 14.

Microsoft told ProPublica that the overall volume of bugs “will not be plateauing for a bit,” but a spokesperson said the company has “invested heavily in both people as well as AI-powered triage solutions that scale quickly to handle the growing number of vulnerabilities.”

Given the new realities of the AI age, including the chaining capabilities, companies like Microsoft might need to rethink their entire approach to triage, said Nguyen, the NSA’s former AI chief. Rather than shunting what are now considered low-risk flaws aside, companies should be dedicating staff to developing and testing patches for the entire spectrum of vulnerabilities, he said. In other words, the cyber ER needs more doctors and nurses treating illnesses that are life-threatening as well as the minor wounds that could later turn deadly.

“There’s no alternative,” Nguyen said. “The patients are coming in fast and furious.”

Microsoft told ProPublica it’s “always going to be reevaluating and considering whether things that were previously lows or moderates be upgraded or thought about differently. With these AI systems, it makes us rethink some of these things. Across the industry, we’re all looking to see how drastic of a change it will be.”

Microsoft’s users may be particularly vulnerable. The popularity of its offerings, used the world over, makes it a frequent and lucrative target for hackers. In addition, many of its products contain “legacy” code. Developed decades ago using now-outdated technology, this code contains unaddressed flaws and contributes to what is known in the industry as “technical debt.”

But the challenge of fixing the flood of newly found bugs also extends to the rest of the software industry, and to open-source software code that is typically free to use and largely maintained by volunteers. Open-source software underpins internet infrastructure and is incorporated into much of the world’s modern technology, including products offered by major tech companies such as Microsoft.

“Nobody has really figured out how to deal with this, and everybody is casting around for what they need to do,” said J. Michael Daniel, a former cybersecurity adviser to President Barack Obama and the president of the Cyber Threat Alliance, a nonprofit organization focused on cybersecurity. “Our tech debt is coming due.”

Ben Edwards, a data scientist who specializes in managing software vulnerabilities, said the software industry was handling an “intense volume even before AI.”

“It was like drinking from a garden hose on the jet setting before, and now it’s like drinking from a fire hose,” Edwards said. “They might have had the teams that could handle that garden hose. Whether they can handle the fire hose is something else.”

Although the volume of vulnerabilities has grown over the years, Microsoft’s internal group responsible for fielding them, the Microsoft Security Response Center, has been perennially understaffed. Even before the crush of AI-identified bugs, the center fielded hundreds or even thousands of reports a month, pushing the group to its limits, ProPublica has reported.

The size of the center reflects Microsoft’s corporate philosophy: Plugging security holes is a cost center, while making new products is a profit center, former employees said. The company is loath to tie up its best engineers with making security patches — a cost center — instead of developing new products and features that will generate profits, ProPublica has reported.

Microsoft told ProPublica that it does not discuss internal staffing decisions but has made investments in recent years to “focus our teams on keeping our customers secure.” The company “continuously evaluates the staffing, processes, and technologies required to support security response and vulnerability management,” a spokesperson said.

According to the slides that accompanied the May internal presentation, Anthropic provided Mythos access to roughly 50 full-time Microsoft employees, with a goal to “harden critical services before publicly available models catch up.” A slide titled “What’s Next” predicted that the Microsoft Security Response Center would see continued case volume “as public tools catch up” to Mythos.

During the May meeting, one staffer appeared to take comfort in the belief that adversaries “don’t have the source code” that such an AI tool would scan for weaknesses. His colleagues, however, quickly corrected him. Portions of Microsoft’s code have, in fact, fallen into hackers’ hands over the years.

“It might not be this week’s source code,” one person said. “But they’ve got source code. It’s out there.”

In a statement to ProPublica, Microsoft downplayed the comment, saying engineers “design our security processes on the expectation that determined adversaries may gain access to code.”

Filed Under: ai, bug hunting, bug patching, cybersecurity, mythos, security, vulnerabilities

Companies: anthropic, microsoft