Skip to main content
Variable-Width Encoding Unicode Standard 17.0 §3.9 RFC 3629

UTF-8 Character Encoding

UTF-8 (8-bit Unicode Transformation Format) is the dominant variable-width character encoding of the modern web and computing systems. It encodes every valid Unicode scalar value (U+0000 to U+D7FF and U+E000 to U+10FFFF) into sequences of one to four 8-bit bytes, maintaining seamless backward compatibility with 7-bit ASCII while strictly prohibiting surrogate code points and overlong byte representations.

Technical Quick Facts

Code Unit Size 8-bit octets (1 byte)
Bytes per Scalar Value 1, 2, 3, or 4 bytes
Permitted Codespace U+0000 to U+10FFFF Surrogates U+D800–U+DFFF strictly excluded
ASCII Compatibility Direct 1:1 Identity Bytes 0x00–0x7F match ASCII identically
Endianness / Byte Order Not Applicable Defined strictly as sequential octets
Byte Order Mark (BOM) Optional (EF BB BF) Neither required nor recommended on Web
Web Ecosystem Share 98%+ Worldwide Universal default per WHATWG Encoding
Governing Specifications Unicode 17.0 / RFC 3629 Obsoletes 5-byte & 6-byte RFC 2279

UTF-8 Bit Layout & Byte Distribution

In accordance with Unicode 17.0 Section 3.9 (Table 3-6), Unicode scalar values are mapped into 1 to 4 bytes using prefix-free binary bit templates. The number of high-order 1 bits in the leading byte directly announces the total length of the sequence, while all continuation bytes begin with the fixed prefix 10xxxxxx.

Length Scalar Value Range Byte 1 (Lead) Byte 2 (Continuation) Byte 3 (Continuation) Byte 4 (Continuation) Lead Byte Hex Capacity
1 Byte U+0000–U+007F 0xxxxxxx — — — 00–7F 128
2 Bytes U+0080–U+07FF 110xxxxx 10xxxxxx — — C2–DF 1,920
3 Bytes U+0800–U+FFFF (excl. surrogates) 1110xxxx 10xxxxxx 10xxxxxx — E0–EF 61,440
4 Bytes U+10000–U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx F0–F4 1,048,576
ℹ️

Prefix-Free Architecture & Self-Synchronization

Because continuation bytes always have bit pattern 10xxxxxx (hex 0x80–0xBF), they can never be confused with single-byte ASCII (0xxxxxxx, hex 0x00–0x7F) or multibyte lead bytes (11xxxxxx, hex 0xC2–0xF4). A stream decoder can pick up at any random byte and reliably seek forward or backward to the next character boundary in at most 3 byte checks without state loss.

Interactive UTF-8 Bit Inspector & Byte Validator

Inspect real-time bit allocations for any text, or validate raw hex byte sequences directly against Unicode 17.0 Table 3-7 constraints.

Preset Examples:
3 Total Bytes
1 Code Point(s)
1 Grapheme Cluster

Encodable Byte Stream

Byte 1 (Lead byte (1110xxxx, indicates 3 bytes))
0xE2
11100010
Dec: 226
Byte 2 (Continuation byte (10xxxxxx))
0x82
10000010
Dec: 130
Byte 3 (Continuation byte (10xxxxxx))
0xAC
10101100
Dec: 172
Hex Sequence: E2 82 AC
Binary Octets: 11100010 10000010 10101100

Detailed Code Point Breakdown

€
EURO SIGN U+20AC (Decimal 8364)
3 Bytes
Scalar Bits (16-bit): 0010 0000 1010 1100
Placed UTF-8 Bits: 11100010 10000010 10101100

Worked Example: Encoding the Euro Symbol (€)

Follow the exact bitwise algorithm mandated by Unicode Standard 17.0 to encode the Euro currency sign € (U+20AC) into its canonical 3-byte UTF-8 representation.

1

Identify Unicode Scalar Value & Determine Byte Length

The Euro sign has scalar value U+20AC (decimal 8,364). Checking Table 3-6:

U+0800 ≤ U+20AC ≤ U+FFFF → Requires a 3-byte UTF-8 sequence.
2

Retrieve Binary Representation & UTF-8 3-Byte Template

Convert 0x20AC into 16 binary bits and prepare the 3-byte prefix template:

Raw Scalar Bits: 0010 0000 1010 1100
3-Byte Template: 1110xxxx 10xxxxxx 10xxxxxx
The template provides exactly 16 variable bit slots (x) matching the 16 bits of the scalar value.
3

Distribute Scalar Bits into the Template Slots

Partition the 16 scalar bits from right to left into 6-bit and 4-bit payloads:

High 4 bits → Byte 1 0010
Middle 6 bits → Byte 2 000010
Low 6 bits → Byte 3 101100
4

Assemble Final Octets & Convert to Hexadecimal

Position Prefix Payload Complete Binary Hexadecimal Decimal
Byte 1 (Lead) 1110 0010 11100010 0xE2 226
Byte 2 (Continuation) 10 000010 10000010 0x82 130
Byte 3 (Continuation) 10 101100 10101100 0xAC 172
5

Validation Check Against Unicode 17.0 Table 3-7

Verify against the normative constraints for E0..EF lead bytes:

✓ First byte is 0xE2 (matches range E1..EC, EE..EF).
✓ Second byte is 0x82 (falls cleanly within valid 80..BF continuation range).
✓ Third byte is 0xAC (falls cleanly within valid 80..BF continuation range).
Result: E2 82 AC (3 bytes, well-formed UTF-8 representation of €).

Representative UTF-8 Character Examples

Explore how different Unicode characters, combining diacritics, and composite sequences map into 1, 2, 3, 4, or multi-code-point byte streams.

A
1 Byte

Latin Capital Letter A

Code Point:
UTF-8 Hex: 41

é
2 Bytes

Latin Small Letter E with Acute

Code Point:
UTF-8 Hex: C3 A9

€
3 Bytes

Euro Sign

Code Point:
UTF-8 Hex: E2 82 AC

∞
3 Bytes

Infinity

Code Point:
UTF-8 Hex: E2 88 9E

中
3 Bytes

CJK Unified Ideograph-4E2D (Zhong)

Code Point:
UTF-8 Hex: E4 B8 AD

ع
2 Bytes

Arabic Letter Ain

Code Point:
UTF-8 Hex: D8 B9

𐂃
4 Bytes

Linear B Ideogram B105 Equus (Horse)

Code Point:
UTF-8 Hex: F0 90 82 83

😀
4 Bytes

Grinning Face

Code Point:
UTF-8 Hex: F0 9F 98 80

Well-Formedness Criteria (Unicode 17.0 Table 3-7)

Not all sequences matching the general bit templates in Table 3-6 are legal UTF-8. To prevent overlong security exploits and surrogate corruption, the Unicode Standard imposes the normative constraints in Table 3-7. Any byte sequence deviating from these exact ranges is ill-formed and must be rejected by compliant decoders.

Scalar Value Range 1st Byte 2nd Byte 3rd Byte 4th Byte Normative Constraint Rationale
U+0000..U+007F 00..7F — — — Standard 7-bit ASCII range.
U+0080..U+07FF C2..DF 80..BF — — C0..C1 prohibited to prevent overlong encoding.
U+0800..U+0FFF E0 A0..BF 80..BF — Second byte starts at A0 to avoid overlong U+0000..U+07FF.
U+1000..U+CFFF E1..EC 80..BF 80..BF — Standard BMP range.
U+D000..U+D7FF ED 80..9F 80..BF — Second byte capped at 9F to exclude surrogate code points (U+D800..U+DFFF).
U+E000..U+FFFF EE..EF 80..BF 80..BF — BMP post-surrogate range.
U+10000..U+3FFFF F0 90..BF 80..BF 80..BF Second byte starts at 90 to avoid overlong BMP encoding.
U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF Planes 4 through 15.
U+100000..U+10FFFF F4 80..8F 80..BF 80..BF Second byte capped at 8F to enforce U+10FFFF maximum scalar limit.

Elimination of Overlong Sequences

In early drafts (and RFC 2044), decoders could accept redundant representations of characters, such as 0xC0 0xAF for / (ASCII 0x2F). Malicious payloads bypassed firewalls by encoding ../ or <script> into 2-byte or 3-byte forms. Under Table 3-7, lead bytes C0 and C1 are universally prohibited, and second bytes for E0 and F0 start strictly at A0 and 90 respectively.

Absolute Surrogate Prohibition

Unicode scalar values are explicitly defined as code points excluding the high and low surrogate range (U+D800 to U+DFFF). Because surrogates exist solely as 16-bit mechanisms for UTF-16, lead byte 0xED restricted second bytes to 80..9F. Second bytes A0..BF would encode surrogates and are strictly illegal in UTF-8.

Ill-Formed Sequences & Decoder Security

Robust security requires decoders to fail safely when encountering non-conformant byte patterns. When decoding untrusted inputs, decoders must never interpret ill-formed sequences as valid characters or silently strip illegal octets.

Test Vector (Hex) Error Classification Targeted Sequence / Value Standard Explanation Security Vulnerability Risk Action
C0 AF OVERLONG_ENCODING ASCII Slash / (U+002F) Illegal 2-byte representation for an ASCII character. Must be encoded in 1 byte (0x2F). Historical security risk in path traversal filter bypasses. Filter evasion
E0 80 AF OVERLONG_ENCODING Non-shortest BMP representation Illegal 3-byte form representing a code point below U+0800. Second byte must be A0–BF when lead byte is E0. Parser confusion
ED A0 80 SURROGATE_ENCODING High Surrogate U+D800 UTF-8 is prohibited from encoding surrogate code points (U+D800–U+DFFF). Surrogates exist exclusively for UTF-16 pair mechanics. Invalid scalar insertion
F4 90 80 80 OUT_OF_RANGE Above Maximum (U+110000) Exceeds the 21-bit Unicode codespace boundary (U+10FFFF). Modern UTF-8 strictly rejects values above U+10FFFF. Memory buffer overflow
E2 82 TRUNCATED_SEQUENCE Truncated Euro Sign Lead byte E2 announces a 3-byte sequence, but stream terminates after 2 bytes. Decoders must emit U+FFFD replacement character. Data truncation
80 INVALID_CONTINUATION Orphaned Continuation Byte Continuation byte 0x80 (10000000) encountered without a preceding lead byte. Stream desynchronization

⚠️ Replacement Character (U+FFFD ) Policy

Per Unicode Standard §5.22 and the WHATWG Encoding Standard, when an ill-formed byte sequence is encountered in text streams, conforming decoders must substitute each maximal subpart of an ill-formed subsequence with exactly one replacement character (U+FFFD REPLACEMENT CHARACTER).

Decoders must never drop invalid bytes silently, as doing so can concatenate previously harmless characters into dangerous exploit strings (e.g. converting s<C0 AF>cript into script).

🔒 Web Application Firewall (WAF) Evasion

Historical vulnerabilities (such as IIS directory traversal CVE-2000-0884) occurred because security filters inspected ASCII representations of strings before normalization, while back-end systems decoded overlong UTF-8 bytes like C0 AF into /.

Strict enforcement of Table 3-7 at the immediate protocol boundary prevents all non-shortest form bypasses completely.

Byte Order Mark (BOM) Semantics in UTF-8

In 16-bit and 32-bit encodings (UTF-16 and UTF-32), the Byte Order Mark (U+FEFF) is necessary to determine processor endianness (Big-Endian vs Little-Endian). In UTF-8, however, code units are strictly 8-bit octets, making byte endianness completely moot.

UTF-8 BOM Hex: EF BB BF
Not Recommended on Web

Where BOM Appears

Some Windows utilities (such as classic Windows Notepad) insert the three bytes EF BB BF at the very beginning of a file as a signature to differentiate UTF-8 from legacy ANSI code pages (like Windows-1252).

Why BOM Causes Failures

  • POSIX Shebang Breaks: A BOM before #!/usr/bin/env prevents the kernel from recognizing executable scripts.
  • JSON Standard Violation: RFC 8259 §8.1 strictly forbids leading BOM in JSON text.
  • HTTP Header Premature Send: In PHP and server scripts, an unnoticed BOM sends output before header() calls, triggering fatal "headers already sent" warnings.
  • Command-Line Tool Mismatches: Unix CLI tools (grep, cat, awk) may treat the BOM as literal content on the first line.
💡

Authoritative Guidance (Unicode & W3C)

Unicode Standard 17.0 §3.10: "Use of a BOM on UTF-8 text is neither required nor recommended for interchange... Systems must not require a BOM on UTF-8 text." Modern web servers and protocols should declare encoding via HTTP headers (Content-Type: text/html; charset=utf-8) or HTML meta tags rather than relying on BOM signatures.

Core Concepts & Engineering Considerations

Mastering UTF-8 requires distinguishing between storage bytes, abstract code points, and user-perceived characters across programming languages.

Fundamental Rule

Byte vs. Code Point vs. Grapheme Cluster

A code point is an abstract Unicode integer (e.g. U+20AC). A byte is an 8-bit storage unit. A user-perceived character (grapheme cluster) can span multiple code points and bytes. Never assume 1 byte equals 1 character.

Interoperability

ASCII Identity Mapping

The first 128 Unicode code points (U+0000 through U+007F) encode into identical single-byte values (0x00 through 0x7F). Any valid 7-bit ASCII stream is identical, valid UTF-8.

Architecture

UTF-8 vs. Unicode

Unicode is the universal repertoire and numbered catalog of characters. UTF-8 is the mathematical serialization algorithm that encodes those numbers into physical bytes.

Storage Standard

UTF-8 BOM Semantics

The byte sequence EF BB BF (UTF-8 encoding of U+FEFF) functions strictly as an optional file format signature. Because UTF-8 operates on single 8-bit octets, endianness is not applicable. UTF-8LE and UTF-8BE do not exist.

Authoritative Standards & Normative Citations

The definitions, bit algorithms, and validation criteria on this page are derived directly from the governing formal specifications.

Unicode Consortium

The Unicode Standard 17.0, Chapter 3 (§3.9) ↗

Defines conformance requirement D92 (UTF-8 encoding form), Table 3-6 (bit distribution), and Table 3-7 (well-formed byte sequences).

IETF

RFC 3629: UTF-8 Transformation Format ↗

Standardizes UTF-8 as STD 63, restricting byte length to 4 octets and requiring shortest-form representations.

WHATWG

Encoding Living Standard (§4.2 UTF-8) ↗

Specifies browser decoding algorithms, error handling, and replacement character (U+FFFD) substitution rules.