Skip to main content
Architectural Switchboard Unicode 17.0 / WHATWG 6 Core Topics 10 Key Distinctions

Character Encodings & Representations

The developer switchboard for binary text encoding forms and source markup escapes. Understand how abstract Unicode code points translate to physical bytes, compare storage trade-offs, and inspect deterministic representations across modern web and OS runtimes.

Encoding Topics & Representation Formats

Direct access to specialized technical guides and specifications. Select a format below to inspect byte layouts, transformation rules, and code examples.

Unicode Encoding Form RFC 3629 / Unicode Standard §3.9

UTF-8 Encoding

Code Unit / Width: 8-bit Code Units (1–4 Bytes)

The universal, variable-width encoding for modern web and software systems. 100% backward-compatible with ASCII, using 1 to 4 bytes per Unicode character.

Unicode Encoding Form RFC 2781 / Unicode Standard §3.8

UTF-16 Encoding

Code Unit / Width: 16-bit Code Units (2 or 4 Bytes)

Variable-width encoding using 1 or 2 16-bit code units. Widely utilized in operating system runtime environments (Windows Win32, Java, JavaScript engines) with surrogate pairs.

Unicode Encoding Form Unicode Standard §3.10

UTF-32 Encoding

Code Unit / Width: 32-bit Code Units (4 Bytes Fixed)

Fixed-width encoding using exactly one 32-bit code unit per Unicode scalar value. Offers constant-time indexing for raw code points, but uses quadruple the memory of ASCII.

Markup Escape Syntax WHATWG HTML Living Standard §13 & §14

HTML Entities & References

Code Unit / Width: ASCII Text Tokens

Source-level markup syntax (€, €, €) used in HTML and XML to author characters without collision with document tags or character set limitations.

URI Octet Escaping IETF RFC 3986 / WHATWG URL Standard

URL Percent-Encoding

Code Unit / Width: %HH Hex Byte Triplets

Byte-level URI escape mechanism replacing non-ASCII or reserved octets with percent-encoded byte triplets (%E2%82%AC). Operates on UTF-8 bytes in modern standards.

Coded Character Set The Unicode Standard (ISO/IEC 10646)

Unicode Code Points

Code Unit / Width: Integer Code Points (U+0000–U+10FFFF)

The universal coded character set numbering layer mapping 149,000+ abstract characters to integer code points across 17 planes, independently of byte storage.

Technical Comparison Matrix

Side-by-side technical comparison of character encoding forms and source-level escape formats across bit depth, byte bounds, and primary system runtimes.

Format Classification Primary Purpose Code Unit Size Storage Bounds Euro (€) Representation Primary System Context Reference
UTF-8 Unicode Encoding Form Universal text storage & network serialization 8-bit Code Unit (1 Byte) 1 to 4 bytes (variable) E2 82 AC 3 bytes Web (HTML/CSS/JS), JSON, APIs, text files, Linux/macOS UTF-8 Reference →
UTF-16 Unicode Encoding Form In-memory string representation in runtime engines 16-bit Code Unit (2 Bytes) 1 or 2 code units (2 or 4 bytes) 20AC 1 unit (2 bytes) Windows Win32, Java, JavaScript strings, C# .NET UTF-16 Reference →
UTF-32 Unicode Encoding Form Constant-time code point array indexing 32-bit Code Unit (4 Bytes) 1 code unit (4 bytes fixed) 000020AC 1 unit (4 bytes) Unix wchar_t, internal parser buffers (rare on web) UTF-32 Details →
HTML Entities Markup Escape Syntax Source-level escaping inside HTML/XML documents ASCII Text Tokens Variable string length € or € 6 to 8 ASCII chars HTML source documents, preventing markup tag collisions HTML Entities Guide →
URL Encoding URI Octet Escaping Transmitting non-ASCII/reserved bytes in URIs %HH Hex Triplets 3 ASCII chars per UTF-8 byte %E2%82%AC 9 ASCII chars HTTP query strings, URI paths, form data submission URL Encoding Guide →

Understanding UTF-32: Fixed Width vs. Memory Trade-Off

UTF-32 (UCS-4) uses exactly one 32-bit (4-byte) code unit for every Unicode scalar value (0x00000000 through 0x0010FFFF). This guarantees constant-time O(1) random access by code point index without surrogate pairs.

Why UTF-32 is rarely used for web interchange: ASCII characters that require only 1 byte in UTF-8 consume 4 full bytes in UTF-32 (a 300% storage expansion). Furthermore, fixed 32-bit units do not guarantee 1 unit equals 1 visible character because complex grapheme clusters (combining accents, flags, skin-toned emoji) still require multiple code points.

Character & Byte Inspector

Inspect live byte representations across UTF-8, UTF-16, UTF-32, HTML entities, and URL percent-encoding. Enter any character or select a curated standard below.

Updates representations in real time across UTF-8 bytes, UTF-16 surrogate pairs, HTML references, and URL encoding.

Presets:

Euro Sign

U+20AC Dec: 8364 Basic Multilingual Plane (BMP, Plane 0)

Standard BMP character demonstrating 3-byte UTF-8, single 16-bit UTF-16 unit, named HTML entity €, and URL percent-encoded bytes.

UTF-8 Byte Sequence 3 Bytes
E2 82 AC
UTF-16 Code Units 1 Unit (2 Bytes)
20AC
UTF-32 Hex 1 Unit (4 Bytes Fixed)
000020AC
HTML Character References Source Escapes
Named: €
Decimal: €
Hex: €
URL Percent-Encoded RFC 3986 Octets
%E2%82%AC
Programming Language Escapes Compiler Literals
JavaScript: \u20AC
Python: \u20AC
CSS: \0020AC

Core Concepts & Terminology Disambiguation

Essential architectural distinctions resolving common industry confusions across Unicode repertoires, code units, surrogate mechanics, escapes, and normalization.

Fundamental Rule Unicode Standard 17.0 §3.4 & §3.9

Unicode ≠ UTF-8: Repertoire vs. Encoding Form

Unicode is the universal numbered catalog of abstract characters (assigning integer code points U+0000 through U+10FFFF). UTF-8 is the mathematical encoding form that translates those code points into 1 to 4 eight-bit bytes for physical storage and network transport.

Key Rule: Unicode defines WHAT the characters are; UTF-8 defines HOW those characters are stored as bytes.
Text Processing Unicode Standard 17.0 §2.4 & UAX #29

Code Point vs. Code Unit vs. Byte vs. Grapheme Cluster

A code point is a Unicode number (e.g. U+1F600). A code unit is the minimal building block of an encoding form (8-bit for UTF-8, 16-bit for UTF-16, 32-bit for UTF-32). A byte is 8 bits. A grapheme cluster is a user-perceived character on screen that can span multiple code points.

Key Rule: Never assume one code unit equals one character. Combining marks, skin-tone modifiers, and ZWJ emoji join multiple code points into a single grapheme cluster.
Surrogate Mechanics RFC 2781 / Unicode Standard §3.8

UTF-16 Surrogate Pairs: One Character, Two Code Units

Characters outside the Basic Multilingual Plane (> U+FFFF, such as emoji and historical scripts) cannot fit in a 16-bit code unit. UTF-16 encodes them as a surrogate pair: a High Surrogate (0xD800–0xDBFF) followed by a Low Surrogate (0xDC00–0xDFFF). Together, the pair represents a single character.

Key Rule: A surrogate pair is NOT two characters. Individual surrogate code points are invalid on their own and cannot appear in valid UTF-8 or UTF-32.
Memory & Indexing Unicode Standard 17.0 §3.10

UTF-32 Fixed Units vs. Grapheme Clusters

UTF-32 assigns exactly one 32-bit code unit per Unicode scalar value. While this makes code point array indexing fast and constant-time, it does not guarantee one 32-bit unit equals one user-perceived glyph because compound grapheme clusters still require multiple code points.

Key Rule: UTF-32 provides fixed-width code points, NOT fixed-width visual characters.
Web Authoring WHATWG HTML Living Standard §13.1.4

HTML Character References Are Markup Escapes, Not Encodings

HTML entities (numeric decimal €, hex €, and named €) are source-code representations interpreted by the HTML parser. The document itself still possesses an underlying character encoding (almost universally UTF-8). Entities allow authoring characters without syntax collisions.

Key Rule: HTML entities exist in HTML source text; they are not an alternative binary encoding format.
URI Protocol RFC 3986 §2.1 & WHATWG URL Standard

URL Percent-Encoding Operates on Bytes, Not Code Points

RFC 3986 percent-encoding replaces disallowed octets with % followed by two hexadecimal digits (%HH). In modern web architectures, Unicode text is first converted to UTF-8 bytes, and each non-ASCII or reserved byte is then percent-encoded (%E2%82%AC for Euro).

Key Rule: URL encoding escapes bytes, not characters. Context also matters: spaces become %20 in URI paths, but application/x-www-form-urlencoded forms often use +.
Equivalence UAX #15: Unicode Normalization Forms

Unicode Normalization (NFC / NFD) Is Not Encoding

Unicode normalization forms (NFC, NFD, NFKC, NFKD) transform equivalent sequences of code points between precomposed single characters (e.g. é U+00E9) and decomposed combining sequences (e.g. e U+0065 + acute accent U+0301).

Key Rule: Normalization changes which code points are used; character encoding determines how those code points become bytes.
Syntax Escapes ECMAScript §12.8.4 & W3C CSS Syntax §4.3

Programming Escapes (\uXXXX) Are Not Encodings

Notations such as \u20AC in JavaScript/Java, \u{20AC} in PHP/Rust, or \0020AC in CSS are compiler string literal escape syntaxes. They instruct the language runtime which code point to instantiate in memory, but are not binary file encodings.

Key Rule: String escapes are source code conveniences; binary encodings are physical byte formats.
Data Transport IETF RFC 4648

Base64 Is Binary-to-Text, Not a Character Encoding

Base64 (RFC 4648) translates arbitrary raw binary octets into a safe 64-character ASCII representation (A–Z, a–z, 0–9, +, /) to prevent transmission corruption across text-only legacy protocols (such as MIME email).

Key Rule: Character encodings turn text into bytes; Base64 turns arbitrary bytes back into ASCII text.
Byte Order Unicode FAQ: UTF-8, UTF-16, UTF-32 & BOM

UTF-8 BOM Is a Signature, Not an Endian Indicator

Because UTF-8 code units are 8-bit octets, byte order (endianness) does not exist in UTF-8. A Byte Order Mark (0xEF 0xBB 0xBF) in UTF-8 acts solely as an encoding signature. Modern web standards strongly discourage UTF-8 BOMs. Endianness (LE vs. BE) applies strictly to UTF-16 and UTF-32.

Key Rule: UTF-8 never requires a BOM. Endianness matters only when code units exceed 8 bits.

Encoding & Conversion Tools

Specialized client and server-side utilities on CopyCharacter for transforming text, escaping entities, and inspecting code points.

Markup Tool

HTML Entity Converter

Encode special characters into HTML named or numeric decimal/hex entities, or decode entity strings back to readable text.

URI Tool

URL Encoder & Decoder

Convert text to RFC 3986 percent-encoded URI strings (or form application/x-www-form-urlencoded), and decode escaped query parameters.

Normalization

Unicode Normalizer

Transform text across Unicode normalization forms (NFC, NFD, NFKC, NFKD) to resolve combining characters and ensure binary equivalence.

Analysis

Unicode Inspector

Analyze strings character-by-character to inspect code points, character categories, blocks, and invisible zero-width characters.

Developer Tool

Unicode Encoder

Convert characters into programming language escape sequences including JavaScript (\uXXXX), Python, CSS, and C/C++.

Developer Tool

Unicode Decoder

Parse raw Unicode escape sequences (\uXXXX, \U0001XXXX, U+XXXX) back into live rendered characters and symbols.

Authoritative Standards & Specifications

All character definitions, byte transformations, and entity representations on CopyCharacter strictly conform to the following international standards.

Unicode Consortium

The Unicode Standard, Version 17.0

Chapter 3 defines conformance and formal Character Encoding Forms (UTF-8 in §3.9, UTF-16 in §3.8, UTF-32 in §3.10).

WHATWG

Encoding Living Standard

Specifies interoperable character encoding algorithms for web browsers, establishing UTF-8 as the mandatory web standard.

IETF

RFC 3629: UTF-8 Transformation Format

Restricts UTF-8 to a maximum of 4 bytes per character, aligning with the 21-bit Unicode codespace (U+0000 to U+10FFFF).

IETF

RFC 2781: UTF-16 Encoding of ISO 10646

Formalizes UTF-16 code unit layout and the mathematical algorithms for computing high and low surrogate pairs.

IETF

RFC 3986: Uniform Resource Identifier (URI)

Defines generic syntax and percent-encoding (%HH) rules for escaping reserved and non-ASCII octets in URIs.

WHATWG

HTML Living Standard §13 & §14

Defines HTML character references (numeric decimal, numeric hex, and the dictionary of 2,231 named character entities).