Encoding Converters Glossary — Every Term Explained

2026-03-27 · SPUNK13 · spunk.bet

Encoding problems are almost always vocabulary problems: someone converts a "character" when they meant a byte, or a "string" when they meant a code point. This glossary defines the terms you actually meet in converter tools, with the command that demonstrates each one.

The character model

• Code point — a number assigned to a character by Unicode, written U+00E9. There are just over a million possible; about 150,000 are assigned.
• Code unit — the fixed-size chunk an encoding uses. UTF-8 uses 8-bit units, UTF-16 uses 16-bit units. One code point can take several code units.
• Grapheme cluster — what a user calls one character. A family emoji can be one grapheme cluster made of seven code points and 25 UTF-8 bytes. This is why len(s) disagrees with what people see.
• Normalisation — é can be one code point (U+00E9) or two (e + combining acute). NFC composes, NFD decomposes. Compare strings only after normalising, or your search will miss matches that look identical.

Encodings you will meet

• ASCII — 7-bit, 128 code points. Every byte below 0x80 in UTF-8 is identical to ASCII, which is why UTF-8 is backwards compatible.
• UTF-8 — 1 to 4 bytes per code point. The default for the web and for anything new.
• UTF-16 — 2 or 4 bytes; characters outside the Basic Multilingual Plane use surrogate pairs. This is what JavaScript and Java strings are made of internally.
• Latin-1 / Windows-1252 — single-byte legacy encodings. Windows-1252 fills Latin-1's control range with smart quotes and em dashes, which is the source of most stray "’" sequences.
• BOM — the byte order mark, EF BB BF in UTF-8. Harmless in a text editor, fatal at the start of a CSV header or a PHP file.

Transfer encodings

• Base64 — turns 3 bytes into 4 ASCII characters, a 33% size increase, padded with =. It is not encryption and not compression.
• Base64url — same alphabet with - and _ replacing + and /, usually unpadded. Used in JWTs and URLs.
• Percent-encoding — URL escaping, %20 for space. Reserved characters differ by URL component, which is why encoding a whole URL in one call breaks it.
• Punycode — encodes internationalised domain names into ASCII with an xn-- prefix. A common phishing vector when lookalike scripts are involved.
• Quoted-printable — an email transfer encoding that keeps text mostly readable and escapes the rest as =XX.

The commands that resolve arguments

file -I document.txt guesses the encoding. iconv -f windows-1252 -t utf-8 in.txt > out.txt converts between them, and iconv -c drops characters that cannot be represented rather than failing. xxd file | head shows you the actual bytes, which settles arguments faster than any discussion. printf 'hi' | base64 and base64 -d round-trip Base64. In Python, 'é'.encode('utf-8') gives you b'\xc3\xa9' — two bytes, one code point, one grapheme.

Mojibake and how to reverse it

Mojibake is text decoded with the wrong encoding: UTF-8 bytes read as Windows-1252 produce "é" where "é" belonged. It is usually recoverable — encode back to the wrong encoding, then decode as the right one — but only if nothing has replaced unmappable bytes with U+FFFD, the replacement character. Once you see "�", the original bytes are gone.