Attackers are targeting open-source AI just as Big Tech is embracing it

The future of AI will be driven in significant part by startups and small players using open-source software to make their own, cheaper, more efficient versions of Claude or ChatGPT, tech leaders finally admitted this week. For companies like AWS, the challenge now is protecting those public tools from increasingly sophisticated attacks from China, Russia, North Korea, or other players.

“Open-weight models—AI models that anyone can download, inspect, modify, and run on their own infrastructure—are an important part of that foundation because they make advanced AI more accessible, adaptable, and widely available,” reads a July 24 statement by Nvidia CEO Jensen Huang, and co-signed by Amazon, Meta, Google, Microsoft, and a host of other companies. 

How open won

The statement shows a dramatic reversal by companies that made fortunes off of proprietary software and have invested billions in OpenAI, Anthropic, and other frontier AI labs whose primary business is closed AI models. Dramatic, yes. But years in the making.

Satya Nadella illustrates how quickly sentiment among leading tech firms has changed. In 2024, the Microsoft CEO described proprietary models as critical to safety. Closed-source AI, he said, “allow[s] us to do deep end-to-end red-teaming, alignment, and safety evaluations before exposing them to the world.” Microsoft had invested $13 billion in OpenAI by that point.

AWS, another signatory of Huang’s statement, this week completed a $50 billion investment in OpenAI; the Amazon spinoff has also put money into rival lab Anthropic. Both of those investments were bets that the intellectual property of a small number of labs was going to be better and easier to safeguard than code made, often, by volunteers.

What has changed since 2024? Several things: A growing body of research showed that relatively cheap open-weight models were steadily catching up to the performance of closed ones. The American public is increasingly pessimistic about AI. The Pentagon wants AI that it can control and can operate without large, targetable data centers. 

But for tech giants like Microsoft and AWS, the biggest change since 2024 is their evolution from provider of AI to provider of tools, space, and security for open-source and open-weight AI developers.

During AWS’ earnings call on Thursday, CEO Andy Jassy boasted of more than 10 models in Amazon’s Bedrock platform, which allows users to build their own generative models. 

Nadella did the same on behalf of Microsoft this week. “We offer the broadest model catalog in the cloud, with over 11,000 models, including the latest from OpenAI, Anthropic, Mistral, xAI,” and Microsoft itself. 

Venture capitalist Chris Dixon in his 2025 book Read, Write, Own describes this phenomenon as “commoditizing the complement.” Big Tech’s recent open-source enthusiasm is not unlike Oracle’s support for the development of the open-source Linux operating system in 2006. Oracle’s strategy, as Dixon describes it, was to get volunteers to make and maintain an operating system that was cheaper than Microsoft Windows. And they did.

Today, Microsoft and AWS see themselves more and more not as the future’s most powerful builders of AI but sellers of tools, services, computing resources, etc., to a wide ecosystem of AI builders—including their own.

Among those services are AI tools like Microsoft Copilot and Amazon Inspector, which help find malware and vulnerabilities in builder code, including code from open-source libraries.

Poisoning the future

AWS said adversaries are increasingly turning to AI not just to find vulnerabilities in open-source code, but to poison those code libraries in ways even other AI security programs don’t detect; for instance, malware that only executes when a user issues a prompt that has a typo or that only works when other code is entered into the library later.

This dangerous code is often cloaked in helpful suggestions. 

“How does that malicious package or software get into that open source?” asked Rick Anthony, Sr., who manages Amazon Inspector. “Attackers are gaining trust… They’re going out and they’re acting like real developers. You know, they’re creating packages, and these packages are doing real useful benefits.”

These techniques are easier to execute with AI coding agents, Anthony said. “They can sit there and have very reasonable-looking contribution histories. They can have very useful release cycles, and before you know it, these attackers look like good citizens within the open-source community.”

Attackers are also exploiting the fact that more and more security reviews now happen via AI agents, which have weaknesses and blind spots. “What we’re going to see is attackers not only try to fool the humans, but try to fool the AI by giving it enough evidence to convince it that what you’re running is ‘OK.’”

China and Russia are in a great position to carry out such attacks because they don’t face penalties for running experiments on real-world targets, AWS Chief Security Officer Stephen Schmidt said. he was He said he is “really concerned” about them.

AWS is employing red teams running with their own AI agents to find vulnerabilities in code before adversaries can exploit them, but also to attack emerging open-weight models, hoping to discover potential adversaries’ tactics before adversaries do.

But finding a hole or vulnerability is only the first part of the challenge, Schmidt said. Developing an actually useful patch takes longer. So AWS is also looking to speed up sending its fixes to the problems it encounters, and then test them again against threats they haven’t yet thought of.

“We test the [patches] for not only performance but also the way that they respond to certain kinds of adverse behavior, because we know that the adversaries are going to go after them as soon as we release them to the public,” Schmidt said.

That, too, will become increasingly difficult for more and more organizations precisely because defenders are now using AI to find more vulnerabilities. Following Anthropic’s release of its powerful Mythos model to a handful of companies, the number of vulnerabilities researchers found and disclosed quickly doubled, as did the number of patches. Each new bug found is a victory, but it also increases the work to develop a good patch.

“We are running this as a security industry as a sprint—oh my gosh! You know this big thing called Mythos came out, and we’ve got to do all this vulnerability identification. This is going to be the long haul. We’re going to be doing this forever” Schmidt said.

The ramifications of that for future software development, and AI model building in particular, are significant. New AI builders will have to invest in protective AI at the same rate they invest in building new tools, he said.

That’s one reason why not everyone is excited about the future of open-source models.

Anthropic is notably absent from Huang’s statement. The company’s CEO issued his own statement this week, stating that he isn’t for banning open-weight models outright, but he supports mandatory safety testing for all models, as well as other measures to curb China’s ability to copy powerful models like Mythos.

Anthropic researcher Julie Merz was more direct about the threat in a Sunday post on X: “This time next year there will be the internet hitting every rural hospital/city council/etc at once with crypto locker attacks,” she said. “I think there’s a shocking lack of imagination in a lot of the CEOs/influencers pushing open models.”