Skip to content
Bamboo Flags
Custom automation & AI for growing businessesDiscover · Design · Deliver · Drive

AI prompts leaked at three providers: what to keep out of them

A security test sent two million AI prompts with fake keys. Attackers breached three AI providers and used the keys. What to keep out of the prompts you send.

Bamboo Flags4 min read

On 19 November 2026, the security company Dragos published the results of a five-month test of AI services. The researchers sent about two million prompts with fake passwords, keys and links in them. Attackers stole prompts from three AI hosting providers and tried the keys. Other services kept or read prompts that were marked "zero data retention".

If your business pastes customer data, passwords or API keys into an AI tool, or runs automations that do it for you, that text can travel further than the tool you use.

What the Dragos test found

From July to November 2026, the Dragos team put unique fake credentials in each prompt. The credentials were canaries: no real system uses them, so any use shows that someone read the prompt. The research is by HD Moore, Chief Research Officer at Dragos. The post does not name the three breached providers. Ars Technica also reported on the research.

The findings:

  • Attackers stole data from three hosting providers. The earliest incident started in mid-July. None of the providers knew about it until the researchers told them.
  • In the largest incident, in late October, attackers took a few hundred fake AWS keys. They made tens of thousands of AWS calls with them, searched for stored secrets, added users and created keys for AI models.
  • Two of the breaches included requests marked zero data retention (ZDR), a setting that tells the provider not to store the prompt.
  • On an AI inference marketplace, individual sellers could read customer prompts through normal request logs. People then tried to log in to the researchers' test servers, one of them with passwords taken from the prompts.
  • A Chinese AI lab fetched the secret links from more than half of the prompts sent to its models from July to October. This included about two-thirds of the requests marked "no data collection" or ZDR. Prompts sent to US providers that resell this lab's models reached the same system.
  • One large US provider still visited a link from a prompt more than two months after the request.

"Zero Data Retention is supposed to mean something. In practice, these prompts were retained, shared with other companies, and read by people."

Dragos also notes that a privacy setting applies only to the first service that receives the request. Each further service on the route can see the full prompt and the full answer.

Together AI: a coding agent leaked the key

On 21 November 2026, Together AI published a notice about prompt leakage. For software testing, the company stored samples of customer prompts in a private repository. The samples came from the MiniMax M3, GLM-5.2 and Kimi K2.6 models, mainly from 24 June, 4 July, 8 July and 14 July 2026.

An AI coding agent put a credential for that repository into a public GitHub project. Together AI's scanning found the credential and the company removed it. Before that, an unauthorized party used it to download the samples, then searched them for credentials to other services and tested them. Five of the tested credentials were Dragos canaries.

Together AI does not say how many customers or requests were affected. It says it will contact affected customers, and that ZDR is now the default for new accounts. For other attacks on AI keys, see our article on stolen AI API keys and LLMjacking.

OpenRouter removed three providers

OpenRouter is a router: one account and one API key that send each request to one of many model providers. On 22 November 2026, OpenRouter published its response. It says the findings showed no breach of OpenRouter itself.

OpenRouter removed three providers from its platform and took 125 endpoints off its ZDR list. It notified all customers who may be affected. The post links to notices from three providers: Novita, Phala and Together AI. It does not name the three providers it removed, and it says that not all affected providers have published a notice.

How Dragos says to choose an AI service

Dragos gives these options:

  • Run the model on your own hardware. No third party reads the prompts, but the cost starts at thousands of dollars.
  • Use a large provider directly. You have one contract and one privacy policy, but breaches and data sharing are still possible.
  • Use a router only with providers that you choose yourself, and check that the privacy setting changes the route.
  • Avoid reseller marketplaces. Dragos says some of them may run on stolen keys.

What to do this week

  1. 01List every AI service your business sends data to.Include the AI steps in Zapier, Make or n8n, and ask each tool vendor which model provider it uses.
  2. 02Remove passwords and API keys from prompts, documents and files that an AI agent can read.An agent sends any credential it can see.
  3. 03Rotate every key or password that was ever pasted into a prompt.Treat it as exposed.
  4. 04Check your OpenRouter or Together AI email for an incident notice.If you use a router, choose the providers yourself. Do not rely on the ZDR setting alone.
  5. 05Put one fake credential into the data your AI automations send.If anyone ever uses it, treat it as a security incident and stop the automation.

Found something worth fixing?

Bring it to a free 45-minute call.

Book a free 45-minute call

More from the blog