82. Unicode

The lesson

Unicode is the universal character standard that assigns a unique number, called a code point, to essentially every character and symbol in use today. A code point is written as U+ followed by four or more hexadecimal digits, for example U+0041 is the Latin capital letter 'A', and U+4E2D is the Chinese character '中' (zhōng). The standard currently defines more than 149,000 characters covering most of the world's writing systems.

The first 128 Unicode code points (U+0000 to U+007F) are exactly the ASCII table. This deliberate design means ASCII text is already valid Unicode text, and it makes migration from the old 7-bit standard trivial. Unicode adds thousands of letters, marks, punctuation, symbols, and emoji beyond that first block.

Unicode itself only assigns numbers; it does not say how those numbers are stored as bytes. The storage scheme is called an encoding. The three common encodings are UTF-8, UTF-16, and UTF-32. UTF-32 uses a fixed 4 bytes for every code point, which is simple but wasteful. UTF-16 uses 2 or 4 bytes and is used internally by Windows and Java. UTF-8 is variable-width and is the dominant encoding on the web and in modern tools.

UTF-8 encodes ASCII characters in a single byte (identical to ASCII), most Latin and Greek letters and Cyrillic in 2 bytes, CJK and most other scripts in 3 bytes, and emoji and rarer characters in 4 bytes. Because it is compact for English and self-synchronising, UTF-8 is the default for JSON, HTML, source code, and most APIs.

This explains a common surprise: an emoji like '😀' (U+1F600) is one character to you but four bytes in UTF-8, while the Chinese character '中' is one character but three bytes. TPEE counts Unicode characters, not bytes, for its 12 prompt sections, so an emoji counts as 1 character even though it uses 4 bytes.

Normalization matters too. Some characters can be written in more than one way — for example 'é' as a single precomposed character U+00E9, or as 'e' (U+0065) followed by a combining acute accent U+0301. Unicode normalization (NFC, NFD, NFKC, NFKD) converts these to a canonical form so that searches and comparisons match correctly.

Why this matters for TPEE: prompts you write may contain accented text, CJK, Arabic, emoji, and other Unicode content. TPEE's font handling and the JSON export must treat these as correct Unicode characters. When you see a garbled 'é' where an 'é' should be, that is usually a UTF-8 vs Latin-1 mismatch, not a data problem. Understanding Unicode helps you write prompts that display and round-trip correctly across languages.

Check yourself