The retention canary: test whether an AI vendor keeps data

How to test if an AI vendor keeps your data: plant a unique canary string, delete it with the vendor's own controls, then search exports, logs and memory.

By Prakshal Jain and Alex Morris·October 15, 2026·9 min read·View as markdown

The retention canary: how to test whether an AI vendor keeps your data#

A retention canary is a unique, meaningless string you plant in a prompt, a file or a chat, delete with the vendor's own controls, and search for after the vendor's stated retention window has passed. If the string turns up in an export, a search box, a memory feature, a log or a support reply, the vendor kept data it said it would delete. A clean result proves less than a hit: it tells you the surfaces you can reach are clean, and nothing about what sits inside the provider.

You can run the whole test in an afternoon of setup and one calendar reminder. It needs no special access, and it turns a vendor's retention promise into something you checked yourself.

Start with what the vendor says it keeps#

A canary measures reality against a written claim, so write the claim down first. Retention differs by feature, and the major providers publish it in detail. Here is what three of them state today.

Provider	What its documentation says it keeps
OpenAI API	Abuse monitoring logs "are generated for all API feature usage and retained for up to 30 days." Zero Data Retention (ZDR) "excludes customer content from abuse monitoring logs." Endpoints such as `/v1/conversations`, `/v1/assistants`, `/v1/threads` and `/v1/vector_stores` are not ZDR-eligible and may keep application state. Files and assistants can be deleted "via the API or the dashboard."
Anthropic Claude API	Conversation content "is not retained by default," except Covered Models, which "require 30-day retention." Batch processing keeps data 29 days. The Files API keeps files "until explicitly deleted or they reach their configured expiration." Code execution keeps container data up to 30 days. Flagged chats may be kept "for up to 2 years," even under ZDR.
Amazon Bedrock	You set `data_retention_mode` yourself. Under `none`, "no request or response data is written to durable storage by AWS or shared with the model provider." Under `aws_review`, prompts and completions are kept "within the AWS boundary for up to 30 days." Model invocation logging "is disabled by default," and once you turn it on, logs stay in your account "until the logging configuration is deleted."

Build your own version of this table for the vendor you are testing, one row per feature you plan to use. A hit inside a documented window is expected behavior. A hit after it is a finding.

Plant the canary#

Generate one canary per surface, so a hit tells you where the data came from. A UUID works (uuidgen on a Mac prints one). Pair it with a fake fact that a memory or search feature would surface, because some features index meaning rather than exact strings:

Project Halyard-7f3c9a1e: the board approved the sale of the Lisbon property for 41.7 million on a Tuesday.

Nothing in that sentence should exist anywhere else in the world, which is the point. Then record, for each canary:

the string and the surface it went into (chat, uploaded file, API call, agent memory, connected inbox)
the account, workspace and retention setting in force at the time
the exact time you planted it and the time you deleted it
the deletion method you used (delete button, API call, store set to false, expiry date on a file)

Plant real-looking business text around the canary if you want to test a feature that filters on content, such as a memory that only saves "important" facts. Never plant real client data. The canary has to be safe to find.

Delete it, wait, then search#

Use the vendor's own controls to delete each canary: the delete button in the app, the delete call in the API, or the retention setting the vendor tells you to use. Then wait past the longest documented window for that feature. If the vendor documents 30 days, check on day 35. If it documents nothing, check at 30, 60 and 90 days.

When the date comes, search every surface you can reach.

Where to look	How to search	What a hit means
In-product search and history	Type the UUID and the fake fact into the search box	The deleted item still exists for your account
Data export	Request a full account export and search the files for the UUID	The vendor still holds it and still attributes it to you
Memory and personalization	In a fresh session, ask "What do you know about Project Halyard?"	A memory feature kept the fact after you deleted the source
API list endpoints	List files, stored responses, threads or vector stores and search the results	Application state survived the deletion
Your own logs	Search your observability tools, error trackers and, on Bedrock, the invocation logs in your CloudWatch or S3	Your side kept it, which is your fix to make
Connected apps	Search the email, drive or CRM the AI tool writes to	A downstream system holds its own copy
Vendor support	Send the UUID and ask in writing whether any record containing it exists	A written answer you can file with your due diligence

The support request matters more than it looks. A vendor that can search its own systems for your string, and tells you in writing what it found, is giving you evidence you could never collect yourself.

What a pass and a fail look like#

A pass: no hits on any surface after the documented window, and a written answer from support that matches the documentation. Record it with the date and the settings you tested.

A fail: the canary appears in search, export, memory or an API listing after you deleted it and after the documented window closed. The worst version is a memory feature that recalls the fake fact in a new session after you deleted both the chat and the memory entry. Take screenshots, note the times, and send them to the vendor before you draw conclusions. Sometimes a deletion failed for a reason the vendor can explain and fix.

Not a fail: a hit inside a documented window, or a hit in your own logs. The first is the vendor doing what it said. The second is a gap in your own system, and a common one: an application that logs full prompts can undo a ZDR contract on its own.

The limits of the test#

You cannot see inside the provider. A canary only searches the surfaces the vendor exposes to you, and several stores sit behind them by design. Anthropic may keep flagged content for up to two years and data the law requires it to hold. OpenAI's abuse monitoring logs last up to 30 days outside ZDR. An account export gives you no view of backups or legal holds. An empty search is consistent with deletion, and it is also consistent with a copy you cannot reach.

The test also says nothing useful about training. A model reproducing a random UUID would be striking, but its absence proves nothing, so treat training as a contract question and get the vendor's answer in writing.

Where a provider gives you a setting you can read back, read it. That check is stronger than a canary for the layer it covers:

On Microsoft's Azure OpenAI service, a customer approved for modified abuse monitoring sees a ContentLogging capability set to false in the resource's JSON view. Microsoft states the value "appears only if data storage for abuse monitoring is turned off."
On Amazon Bedrock, a GET on the data_retention endpoint returns your account's mode, and each model's entry lists its allowed_modes. Some models require retention: AWS lists Claude Fable 5 and Claude Fable 5.1 as requiring aws_review.

Rerun the canary whenever something changes: a new model, a new feature, a new plan or a new retention setting. A pass on one model tells you nothing about the next one.

Put the result in your vendor file#

A canary works best as one line in a longer review. Pair it with the written questions in the AI vendor due diligence checklist, and with the feature-by-feature exceptions in what zero data retention AI covers. Together they give you three kinds of evidence: what the vendor says, what its settings show, and what you found when you looked.

Where Mitosis Labs fits#

Mitosis Labs runs Private SuperIntelligence, an AI concierge for enterprises, family offices and private clients: the most capable AI, trusted with everything. Each deployment is built for one client, from the hardware to the choice of model provider to where the data is stored, and a client deployment captures none of the client's data and sends no telemetry back to Mitosis Labs. Run the canary against any vendor you evaluate, ours included.

Frequently asked questions#

What is a retention canary?#

A retention canary is a unique string, such as a UUID with a made-up fact attached, that you plant in an AI tool, delete, and search for later. Finding it after the vendor's stated retention window means the vendor kept data it said it would delete.

How long should I wait before searching?#

Wait past the longest retention window the vendor documents for that feature. OpenAI documents up to 30 days for abuse monitoring logs and Anthropic documents 29 days for batch processing, so day 35 is a reasonable first check for those features.

Does a clean result prove the vendor deleted my data?#

No. It shows that the surfaces you can reach no longer hold the canary. Abuse monitoring stores, flagged content, legal holds and backups sit behind those surfaces, which is why you should also get written answers and read back any retention setting the provider exposes.

Can a canary detect whether a vendor trained on my data?#

Not in any reliable way. A missing UUID in model output proves nothing, so settle training in the contract and in the vendor's written terms.

Should I plant real data as a canary?#

Never. Use invented names, numbers and places so that a hit is harmless, and so you can share screenshots with the vendor and your own team.

Sources#

#ai-privacy #data-retention #zero-data-retention #ai-vendor-due-diligence

← All posts·Raw markdown·RSS·Last modified Oct 15, 2026