Unicode Never Forgets, and That's a Security Problem

There's a kanji on your phone right now that means nothing. Not "obscure," not "archaic" — nothing. 彁 (U+5F41) has no pronunciation, no definition, and no recorded use in any Japanese text before it appeared in a character encoding standard in 1978. It exists because a standards committee half a century ago made a transcription error, and every layer of infrastructure since has faithfully preserved the mistake. It renders in your terminal, passes your validators, and travels through your APIs like any legitimate character.

The story of how it got there is a good yarn. The lesson it teaches about input validation is better, and most security tooling still hasn't learned it.

Twelve characters nobody ordered

When Japan's standards body assembled JIS X 0208 — the 6,355-kanji repertoire that underpinned Japanese computing for decades — the committee worked from paper sources: government gazetteers, name registries, earlier character lists. The work was literal cut-and-paste. Characters that couldn't be typeset were assembled by hand from pieces, glued down, and photocopied.

Twelve of the resulting kanji, the so-called ghost characters (幽霊文字), turned out to have no source at all. In 1997, a team led by researcher Hiroyuki Sasahara traced each one for the standard's fourth revision, interviewing surviving catalogers and hunting down the original documents. The findings are wonderful. 妛 (U+599B) was supposed to be a genuine place-name character with 山 over 女 — but the two parts had been glued onto paper separately, and the photocopied seam between them read as an extra stroke. The seam got standardized; the real character, 𡚴, didn't enter Unicode until years later. Only 彁 resisted explanation entirely, most likely a misreading of 彊.

Here's the part that matters for anyone who ships software: when the errors were discovered, nobody removed them. Documents in the wild might contain them; round-trip compatibility won. Then CJK unification carried the ghosts into Unicode, whose stability policy is explicit — once a character is encoded, it will never be removed. The photocopier seam of 1978 is now a permanent, load-bearing part of the world's text infrastructure.

An append-only ledger of mistakes

Unicode is best understood as an append-only log. As of Unicode 17.0, released last September, it contains 159,801 characters across 172 scripts, with CJK ideographs alone now past 100,000. Every erroneous character from every national standard it absorbed is still in there, alongside bidirectional control codes, invisible joiners, variation selectors, and a Tags block of invisible codepoints originally meant for language tagging.

No human holds that repertoire in their head. The ghost characters prove that even the committees who encode this stuff don't fully know what's in it — it took a multi-year forensic investigation to explain twelve characters out of six thousand. Yet an enormous amount of security tooling is built on the opposite assumption: that someone, somewhere, can enumerate the dangerous characters and blocklist them.

That assumption keeps failing in the same shape. The Trojan Source attacks (CVE-2021-42574 and CVE-2021-42694, disclosed by Nicholas Boucher and Ross Anderson in 2021) used bidirectional override characters to make source code read one way to human reviewers and compile another way entirely — the controls had been sitting in Unicode for decades, doing legitimate work for Arabic and Hebrew text, before anyone weaponized them in code review. IDN homograph attacks did it earlier with Cyrillic lookalikes in domain names. The current frontier is LLM applications, where invisible Tags-block codepoints have been used to smuggle instructions into prompts that no human reviewer can see. Different decade, same root cause: a corner of the repertoire that validation logic didn't know existed.

Validate the way the ghosts teach you to

The practical upshot isn't "Unicode is scary." It's that any defense phrased as "reject the bad characters" is structurally unsound, because the set of characters — let alone the bad ones — isn't knowable the way a blocklist needs it to be. The defenses that actually hold are phrased the other way around:

• Allowlist by script and property, not by codepoint. "Identifiers may contain Latin letters, digits, and hyphen" survives every future Unicode release. A blocklist of known-bad codepoints is stale the day it ships. UTS #39, Unicode's own security spec, gives you restriction levels and a confusable-skeleton algorithm for detecting lookalike identifiers — use it instead of reinventing it badly.
• Normalize once, at the trust boundary, then never again. Validate the NFC form and store exactly what you validated. Spotify learned this in 2013 when a canonicalization function that wasn't idempotent let attackers hijack accounts by registering Unicode variants of existing usernames — the same class of bug as the ghost characters: a transformation applied one more time than anyone accounted for.
• Reject invisible and directional controls where they have no business. Format-category characters don't belong in usernames, identifiers, or diffs. This one is now table stakes in the toolchain — rustc has denied bidi controls in source since 1.56.1, GCC 12 added -Wbidi-chars, and GitHub flags them in pull requests. If your CI predates 2021 conventions, add the check.
• Stop trusting "it looks like valid text" heuristics. 彁 is the proof case: it passes every well-formedness check, every "is this real Japanese" filter, every human glance — and means nothing. Well-formed and meaningful are independent properties. Validation can only ever enforce the first.

The ghost characters themselves are harmless — twelve inert curiosities that mostly show up in blog posts and font-rendering tests. But they're the cleanest existence proof we have that the text layer contains things nobody put there on purpose and nothing will ever take away. Every security review that treats Unicode as a known quantity — a fixed alphabet with a few weird characters to filter — is repeating the 1978 committee's mistake: assuming the character set is what it appears to be. It never was. Build your boundaries accordingly, and the ghosts stay a good story instead of a postmortem.