Fireworks AI Breach Exposed Notion AI Chat Transcripts, Second Incident in Two Months

Fireworks said in a security advisory that the attacker first breached the company's cloud provider and used that access to reach a Fireworks development system, from which some customer data could be accessed. It is the second security incident at the company in as many months, following one in October.

The breach matters well beyond one startup. Inference providers such as Fireworks sit underneath many of the AI features that software companies sell to their own users, so an intrusion at one supplier can reach the customers of its customers. Fireworks says it has tens of thousands of customers, including legal technology company Harvey and coding tool maker Cursor, according to TechCrunch.

Notion, the online document editor and note-taking app, said in a disclosure that around 1% of its customers were affected. The company said Fireworks notified it of a security incident and later confirmed that Notion customer data had been taken. On November 3, Notion said, it cut all connections between its systems and Fireworks and deleted the stored data.

This week Notion said the stolen material included chat transcripts from Notion AI customers who used its Insights feature. According to the company, a transcript contains "any input to or output from the model that was written to the chat". Notion added that much of the data could not be attributed, meaning it could not determine which customer a given record came from.

The disclosure also raised a question about what Fireworks was keeping. In a note to customers, Notion suggested that although Fireworks had promised zero data retention, meaning it would not store Notion data after processing a request, Notion later learned that data was in fact being retained, The Information reported. The Information also said the breach had prompted some large customers to consider dropping the service.

An expanded Fireworks disclosure on November 9 said the intrusion into the cloud provider hosting its development machines took place in late October, and that for some accounts the exposed material went beyond billing data to secrets, datasets and incomplete model weights, according to a report by Runtime Wire. Fireworks has not publicly named the cloud provider.

Fireworks said it believes the attackers are financially motivated and are mainly interested in selling access to large language models. Such schemes resell stolen access to unauthorised third parties, who can then consume AI model usage paid for by someone else.

For companies that build AI features on third-party inference, the incident puts zero-retention promises and supplier security reviews under scrutiny. Fireworks is one of the most heavily funded startups among inference providers, a group that helps developers run open-source models at high speed on Nvidia servers.

Fireworks has not said how many of its customers were affected in total, and Notion is so far the only customer to confirm publicly that its users' data was taken.