Skip to main content
16-Bit Variable-Width Encoding Unicode Standard 17.0 §3.8 & §3.9 RFC 2781

UTF-16 Encoding

UTF-16 (16-bit Unicode Transformation Format) is the variable-width character encoding that represents Unicode scalar values using either one or two 16-bit code units (2 or 4 bytes). Characters within the Basic Multilingual Plane (U+0000 to U+D7FF and U+E000 to U+FFFF) map directly to a single 16-bit unit. Characters in supplementary planes (U+10000 to U+10FFFF)—including emojis, mathematical alphanumerics, and historic scripts—are encoded using an ordered pair of reserved code units called a surrogate pair.

Technical Quick Facts

Code Unit Size 16-bit Code Units (2 Bytes) Fundamental unit of the encoding form
Units per Scalar Value 1 or 2 Code Units (2 or 4 Bytes) 1 unit for BMP; 2 units for Supplementary
Basic Multilingual Plane (BMP) U+0000–U+D7FF & U+E000–U+FFFF Direct 1:1 scalar mapping (1 unit, 2 bytes)
Reserved Surrogates U+D800–U+DFFF 2,048 units reserved; not scalar characters
Supplementary Planes (1–16) U+10000–U+10FFFF Encoded as Surrogate Pairs (2 units, 4 bytes)
Endianness / Byte Order UTF-16BE / UTF-16LE Big-Endian or Little-Endian serialization
Byte Order Mark (BOM) Optional (U+FEFF) BE: FE FF | LE: FF FE
Standards Baseline Unicode 17.0 §3.8, §3.9 Table 3-5 / RFC 2781 Unicode Consortium & IETF Standard

UTF-16 Code Space & Range Distribution

The Unicode Standard partitions the entire $1,114,112$ code point space into planes. UTF-16 handles code points using single 16-bit code units for BMP scalar characters and pairs of 16-bit units for supplementary characters, in accordance with Unicode Standard 17.0 Table 3-5.

Character Plane Scalar Range Code Units Byte Length Code Unit Range (Hex) Encoding Status
Basic Multilingual Plane (BMP, Lower)
Covers ASCII, Latin, Greek, Cyrillic, Hebrew, Arabic, and most common scripts.
 U+0000–U+D7FF 1 Unit 2 Bytes 0000–D7FF Single Code Unit (Direct 1:1 scalar mapping)
High-Surrogate Range (Lead Surrogates)
Permanently reserved for the leading unit of a surrogate pair. Cannot stand alone.
 U+D800–U+DBFF 1 Unit 2 Bytes D800–DBFF Reserved High Surrogates (Not Scalar Values)
Low-Surrogate Range (Trail Surrogates)
Permanently reserved for the trailing unit of a surrogate pair. Cannot stand alone.
 U+DC00–U+DFFF 1 Unit 2 Bytes DC00–DFFF Reserved Low Surrogates (Not Scalar Values)
Basic Multilingual Plane (BMP, Upper)
Private Use Area, CJK compatibility, alphabetic presentation forms, specials.
 U+E000–U+FFFF 1 Unit 2 Bytes E000–FFFF Single Code Unit (Direct 1:1 scalar mapping)
Supplementary Planes (Planes 1–16)
Contains emojis, historic scripts (Linear B, Egyptian Hieroglyphs), musical notation, rare CJK ideographs.
 U+10000–U+10FFFF 2 Units 4 Bytes D800 DC00 – DBFF DFFF Surrogate Pair Required (High Surrogate + Low Surrogate)

The Surrogate Pair Mechanism

Because a single 16-bit code unit can address at most $65,536$ values, Unicode reserved $2,048$ code units in the Basic Multilingual Plane (Plane 0) specifically to encode the $1,048,576$ supplementary characters (Planes 1–16).

Anatomy of a Surrogate Pair (Example: 😀 U+1F600) 2 Code Units = 4 Bytes = 1 Unicode Character
High Surrogate (Lead)
D83D
Range: 0xD800–0xDBFF
Encodes High 10 Bits ($0\text{x}03\text{D}$)
+
Low Surrogate (Trail)
DE00
Range: 0xDC00–0xDFFF
Encodes Low 10 Bits ($0\text{x}200$)
=
Decoded Unicode Character
😀
U+1F600
Grinning Face (1 Scalar Value)

Crucial Rule: Neither 0xD83D nor 0xDE00 is a character on its own. They are surrogate code points reserved exclusively to combine into U+1F600.

1. Encoding Formula (Scalar to Surrogates)

For any supplementary code point $C$ where $0\text{x}10000 \le C \le 0\text{x}10\text{FFFF}$:

U' = C - 0x10000;
HighSurrogate = 0xD800 + (U' >> 10);
LowSurrogate = 0xDC00 + (U' & 0x3FF);

Subtracting $0\text{x}10000$ yields a 20-bit number ($2^{20} = 1,048,576$). Splitting into two 10-bit halves fits perfectly into the two 1,024-unit surrogate windows.

2. Decoding Formula (Surrogates to Scalar)

Given a valid high surrogate $W_1$ and low surrogate $W_2$:

C = 0x10000 + ((W1 - 0xD800) << 10) + (W2 - 0xDC00);

Reverses the 10-bit shifts and adds the $0\text{x}10000$ offset to reconstruct the exact 21-bit Unicode scalar value.

Interactive UTF-16 Inspector & Validator

Explore live UTF-16 code unit encoding, toggle between Big-Endian and Little-Endian byte serializations, or input raw hexadecimal code units to validate sequence conformance.

Quick Presets:
Display Format:
Code Points: 1
UTF-16 Code Units: 2
Serialized Bytes: 4
Grapheme Clusters: 1
Hexadecimal UTF-16 Code Units:
D83D DE00
😀 U+1F600
Supplementary Multilingual Plane Surrogate Pair Required
High Surrogate: D83D
Low Surrogate: DE00
BE Bytes: D8 3D DE 00
LE Bytes: 3D D8 00 DE

Worked Example: Encoding 😀 (U+1F600)

Follow the exact mathematical and bit-shifting derivation defined by Unicode Standard 17.0 to convert the supplementary scalar value U+1F600 into its UTF-16 surrogate pair D83D DE00.

Step 1
Subtract 0x10000
0x1F600 - 0x10000 = 0x0F600
Subtracting the base 0x10000 reduces the code point to a 20-bit value (0x00000..0xFFFFF).
Step 2
Extract High & Low 10 Bits
High 10 bits: 0x03D (0000111101) | Low 10 bits: 0x200 (1000000000)
Split the 20-bit offset into two 10-bit values (0 to 1023 each).
Step 3
Add High Surrogate Base (0xD800)
0xD800 + 0x03D = 0xD83D
The high (leading) surrogate unit is computed by adding the top 10 bits to 0xD800.
Step 4
Add Low Surrogate Base (0xDC00)
0xDC00 + 0x200 = 0xDE00
The low (trailing) surrogate unit is computed by adding the lower 10 bits to 0xDC00.
Step 5
Surrogate Pair Result
D83D DE00
Together, the ordered pair <0xD83D, 0xDE00> represents U+1F600 in UTF-16.

Verification by Reverse Decoding

C = 0x10000 + ((0xD83D - 0xD800) << 10) + (0xDE00 - 0xDC00)
C = 0x10000 + (0x03D × 1024) + 0x200 = 0x10000 + 0xF400 + 0x200 = 0x1F600 (U+1F600) ✓

The reverse formula reconstructs the original scalar value with 100% mathematical fidelity.

Representative Examples Across Planes

Representative Unicode characters, mathematical symbols, historic scripts, combining marks, and complex emoji sequences encoded in UTF-16.

A

Latin Capital Letter A

ASCII / Basic Latin
UTF-16 Code Units: 0041
BE Serialized Bytes: 00 41
LE Serialized Bytes: 41 00
1 Code Unit 2 Bytes 1 Code Point

1 code unit (2 bytes). Illustrates 16-bit BMP encoding where high byte is 0x00.

é

Latin Small Letter E with Acute

Latin-1 Supplement
UTF-16 Code Units: 00E9
BE Serialized Bytes: 00 E9
LE Serialized Bytes: E9 00
1 Code Unit 2 Bytes 1 Code Point

1 code unit (2 bytes). Precomposed BMP accented character.

€

Euro Sign

Currency Symbols
UTF-16 Code Units: 20AC
BE Serialized Bytes: 20 AC
LE Serialized Bytes: AC 20
1 Code Unit 2 Bytes 1 Code Point

1 code unit (2 bytes). Common BMP business character (0x20AC).

∞

Infinity

Mathematical Operators
UTF-16 Code Units: 221E
BE Serialized Bytes: 22 1E
LE Serialized Bytes: 1E 22
1 Code Unit 2 Bytes 1 Code Point

1 code unit (2 bytes). BMP mathematical symbol (0x221E).

😀

Grinning Face

Emoji / SMP
UTF-16 Code Units: D83D DE00
BE Serialized Bytes: D8 3D DE 00
LE Serialized Bytes: 3D D8 00 DE
2 Code Units 4 Bytes 1 Code Point

2 code units (4 bytes). Canonical emoji surrogate pair: D83D DE00.

𐂃

Linear B Ideogram B105F Mare

Historic Scripts (SMP)
UTF-16 Code Units: D800 DC83
BE Serialized Bytes: D8 00 DC 83
LE Serialized Bytes: 00 D8 83 DC
2 Code Units 4 Bytes 1 Code Point

2 code units (4 bytes). Tests high surrogate lower boundary (0xD800).

𪚥

CJK Unified Ideographs Extension B

Rare CJK Ideograph (SIP)
UTF-16 Code Units: D869 DEA5
BE Serialized Bytes: D8 69 DE A5
LE Serialized Bytes: 69 D8 A5 DE
2 Code Units 4 Bytes 1 Code Point

2 code units (4 bytes). Plane 2 Ideograph: U+2A6A5 -> D869 DEA5.

é

Combining E + Acute Accent

Combining Sequence
UTF-16 Code Units: 0065 0301
BE Serialized Bytes: 00 65 03 01
LE Serialized Bytes: 65 00 01 03
2 Code Units 4 Bytes 2 Code Points

2 code points, 2 code units (4 bytes). 1 perceived glyph = 2 code units.

👩‍💻

Woman Technologist

ZWJ Emoji Sequence
UTF-16 Code Units: D83D DC69 200D D83D DCBB
BE Serialized Bytes: D8 3D DC 69 20 0D D8 3D DC BB
LE Serialized Bytes: 3D D8 69 DC 0D 20 3D D8 BB DC
5 Code Units 10 Bytes 3 Code Points

3 code points, 5 UTF-16 code units (10 bytes). Complex sequence joined by ZWJ.

Byte Order & Serialization: UTF-16BE vs. UTF-16LE

UTF-16 defines an encoding form using 16-bit code units. When these 16-bit integers are serialized into an 8-bit byte stream (for network transmission or storage), the order of the two bytes within each code unit is governed by the encoding scheme: Big-Endian or Little-Endian.

Character Code Point 16-Bit Code Units UTF-16BE (Big-Endian) UTF-16LE (Little-Endian) Serialization Behavior
A U+0041 0041 00 41 41 00 ASCII: Big-Endian places 00 first; Little-Endian places 41 first.
€ U+20AC 20AC 20 AC AC 20 Euro Sign: Single code unit 20AC inverted in Little-Endian.
∞ U+221E 221E 22 1E 1E 22 Infinity: Single code unit 221E inverted in Little-Endian.
😀 U+1F600 D83D DE00 D8 3D DE 00 3D D8 00 DE Emoji: Note that each 16-bit code unit is byte-swapped individually!
𐂃 U+10083 D800 DC83 D8 00 DC 83 00 D8 83 DC Historic script: D800 DC83 serialized to 00 D8 83 DC in Little-Endian.

Important Note on Surrogate Swapping

When serializing a surrogate pair in Little-Endian mode (UTF-16LE), each 16-bit code unit is byte-swapped individually. The order of the two code units remains unchanged: the High Surrogate always precedes the Low Surrogate. For example, D83D DE00 becomes 3D D8 00 DE, NOT 00 DE 3D D8.

Byte Order Mark (BOM) Semantics

The Byte Order Mark (BOM) is the Unicode character U+FEFF (ZERO WIDTH NO-BREAK SPACE). Placed at the beginning of a serialized UTF-16 stream, its byte pattern immediately reveals whether the stream is Big-Endian or Little-Endian.

UTF-16BE Signature
FE FF

High-order byte 0xFE precedes low-order byte 0xFF. Signals Big-Endian byte order.

UTF-16LE Signature
FF FE

Low-order byte 0xFF precedes high-order byte 0xFE. Signals Little-Endian byte order.

Stream Context Requirement Status Standards Explanation
Explicitly Labeled Streams (e.g. charset=utf-16le or utf-16be) Optional / Not Required When the MIME type or transport layer explicitly declares the endianness, the byte order is already known.
Generic "UTF-16" Byte Streams (without BE/LE designation) Recommended for Detection If labeled simply "UTF-16", an initial BOM identifies whether the stream is Big-Endian (FE FF) or Little-Endian (FF FE).
Generic "UTF-16" Streams Lacking a BOM Default to Big-Endian Per RFC 2781 Section 4.3, in the absence of a BOM, systems without prior agreement should interpret generic UTF-16 as Big-Endian (UTF-16BE).
Mid-Stream Occurrence of U+FEFF Treated as Zero-Width No-Break Space If U+FEFF appears after the stream header, it is treated as a non-breaking space character, though U+2060 (Word Joiner) is preferred.

Universal Requirement Myth

UTF-16 does NOT universally require a BOM. While common in Windows text files (such as Notepad .txt files labeled "Unicode"), protocols specifying charset=utf-16le or charset=utf-16be explicitly do not require a BOM. In web applications, the WHATWG Encoding Standard standardizes on UTF-8, but decodes labeled UTF-16 streams according to explicit BOM or MIME definitions.

Well-Formedness & Ill-Formed Surrogate Sequences

A UTF-16 code unit sequence is well-formed if and only if every surrogate unit appears as part of a correctly paired, ordered sequence. Any isolated or mismatched surrogate violates Unicode Standard 17.0 §3.9 (Definition D92).

Unicode Conformance Invariants

  • Rule 1 (BMP Scalars): Single 16-bit code units must fall in U+0000..U+D7FF or U+E000..U+FFFF.
  • Rule 2 (High Surrogates): Any high surrogate (0xD800..0xDBFF) must be immediately followed by a low surrogate (0xDC00..0xDFFF).
  • Rule 3 (Low Surrogates): Any low surrogate (0xDC00..0xDFFF) must be immediately preceded by a high surrogate (0xD800..0xDBFF).
  • Rule 4 (Noncharacters): Noncharacters (such as U+FDD0..U+FDEF, U+FFFE, U+FFFF) are valid Unicode scalar values and must never be treated as surrogate errors.
Code Units (Hex) Error Classification Error Offset Standard Conformance Diagnostic Security & Parser Risk
D800 UNPAIRED_HIGH_SURROGATE Position 0 Isolated High Surrogate at end of sequence with no matching low surrogate. String truncation vulnerability; decoders may discard or generate replacement characters (U+FFFD).
D83D 0041 EXPECTED_LOW_SURROGATE Position 1 High surrogate (D83D) followed by ASCII letter A (0041) instead of a low surrogate (DC00..DFFF). Surrogate desynchronization; downstream parsers may misinterpret payload boundaries.
DC00 UNPAIRED_LOW_SURROGATE Position 0 Isolated Low Surrogate encountered without a preceding high surrogate. Orphaned trailing surrogate; causes malformed string crashes in strict environments.
DC00 D83D UNPAIRED_LOW_SURROGATE Position 0 Inverted surrogate order: low surrogate (DC00) appears before high surrogate (D83D). Surrogate inversion produces two ill-formed code units; rejected by standard decoders.
D800 D800 EXPECTED_LOW_SURROGATE Position 1 Two consecutive high surrogates; the first high surrogate is never completed. Surrogate stacking error; common when slice operations cut strings through surrogate boundaries.
0041 D83D TRUNCATED_SURROGATE_PAIR Position 1 Valid ASCII A (0041) followed by high surrogate D83D truncated before its low surrogate. Buffer cutoff at surrogate boundary.

Core Concepts & Architecture

Essential distinctions necessary for correctly implementing and debugging UTF-16 strings in modern software.

Code Unit vs. Code Point vs. Byte

In UTF-16, a code unit is 16 bits (2 bytes). A code point is a numeric index in the Unicode codespace. In the BMP, 1 code point = 1 code unit = 2 bytes. In supplementary planes (e.g. 😀), 1 code point = 2 code units = 4 bytes.

User-Perceived Characters & Grapheme Clusters

A single visible character on screen can require more than one code point. For example, 👩‍💻 (Woman Technologist) consists of 3 code points (Woman U+1F469, ZWJ U+200D, Laptop U+1F4BB) and spans 5 UTF-16 code units (10 serialized bytes). Do not confuse code unit counts with human-perceived characters.

UTF-16 vs. UCS-2

UCS-2 was a legacy fixed-width 16-bit encoding that only supported the Basic Multilingual Plane (U+0000..U+FFFF). UCS-2 had no concept of surrogate pairs and could not represent characters beyond 0xFFFF. UTF-16 superseded UCS-2 by introducing surrogate pairs, enabling representation of all 1,114,112 Unicode code points.

UTF-16 vs. UTF-8

UTF-8 is an 8-bit byte-oriented variable-width encoding (1 to 4 bytes per scalar) that is strictly ASCII-compatible. UTF-16 is a 16-bit code-unit variable-width encoding (2 or 4 bytes per scalar). UTF-8 dominates the web, while UTF-16 is widely used as an in-memory string representation in operating systems and runtimes.

Programming String Length Pitfall (JS, Java, C#)

Many programming language string implementations (such as JavaScript "string".length, Java String.length(), and C# string.Length) count 16-bit UTF-16 code units rather than Unicode code points. Consequently, "😀".length reports 2 instead of 1. Use Array.from(str).length or code-point-aware iterators to count true characters.

Authoritative Standards & Normative References

All data, surrogate calculations, well-formedness rules, and byte serialization orders on this page are derived directly from normative technical standards:

  • The Unicode Standard, Version 17.0 — Section 3.8 (Surrogates) & Section 3.9 (Unicode Encoding Forms - Table 3-5) (Unicode Consortium)
  • RFC 2781 — UTF-16, an encoding of ISO 10646 (Internet Engineering Task Force (IETF))
  • WHATWG Encoding Standard — Section 4.5 (utf-16be) & Section 4.6 (utf-16le) (Web Hypertext Application Technology Working Group (WHATWG))