Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification

2026-09-01

Authors: Yisen Xi

ArXiv: 2608.31142v1

PDF: Download PDF

Over the past year, AI labs have gotten into the habit of quietly dropping frontier models onto developer platforms under codenames — think "stealth-mode" releases where nobody officially says which company built the thing you're talking to. That's fun for benchmark leaderboards, but it's a real problem for anyone building on top of these models. Who actually made the model determines where your prompts get logged, what jurisdictions apply, what the pricing will settle at, and whether the vendor is going to yank the endpoint next month. Practitioners have been reduced to folk methods: ask the model who it is, look at its tokenizer quirks, check whether it refuses the same things ChatGPT refuses. None of these have been rigorously tested for accuracy.

This paper proposes a proper forensic protocol for figuring out who built an anonymous API-served model, treating the problem the way a lab would treat identifying an unknown chemical sample. The author lays out a four-stage audit:

The key insight is that self-identification is fundamentally untrustworthy — a model can be system-prompted to deny being what it is, or trained to claim to be something else — but the low-level artifacts of how it was built are much harder to fake. Tokenizers leave characteristic byte-level footprints. Safety training produces recognizable refusal grammars. Serving infrastructure has latency signatures. Together these form a fingerprint that's expensive for a vendor to actively spoof.

Why is this a big deal? Because the "stealth release" model is becoming a real market practice, and the downstream users of these endpoints — startups, enterprises, researchers — currently have no defensible way to say "we know what we're building on." That has consequences for compliance (GDPR data-handling terms depend on the processor's identity), for security (supply-chain risk depends on knowing your vendor), and for reproducibility in research. A validated audit protocol turns a hand-wavy vibe check into something you could put in front of a procurement or legal team.

Why it matters: As AI vendors increasingly ship frontier models anonymously, this protocol gives buyers, regulators, and researchers a defensible method to know what they're actually running against — closing a real accountability gap in the model marketplace.

All newsletters