What Is Unicode?

What problem does Unicode solve?

Before Unicode became the common text standard, many systems used separate character sets and code pages. A byte value that meant one character on one computer could mean a different character on another. Text copied from an old database, email program or document editor could arrive as unreadable symbols because the receiving system guessed the wrong character set.

This was especially painful for multilingual text. ASCII handled basic English letters, digits and punctuation, but it could not represent Greek, Cyrillic, Arabic, Han characters, accented Latin text or emoji. Different regions solved that limitation with local encodings, but those encodings were not enough for a web page, document or database field that needed several languages at once.

Unicode solves the problem by giving software a shared repertoire of characters and a stable number for each one. It is not a font, and it does not decide the exact shape of a letter on screen. Fonts draw glyphs. Unicode defines the characters that software stores, exchanges, searches and compares.

How Unicode represents characters

Unicode describes abstract characters and assigns code points to them. That sounds formal, but the practical idea is simple: software needs a stable way to say “this is LATIN CAPITAL LETTER A” or “this is GRINNING FACE” before a font can draw it or an encoding can store it as bytes.

Characters

A character is the abstract unit of text, such as A, é, Ж, 字 or 😀. A glyph is the visual drawing supplied by a font. The same character can have different glyphs in different typefaces. An encoded value is the byte sequence used to store that character in a format such as UTF-8.

Code points

A code point is the unique number assigned by Unicode. It is commonly written as U+ followed by hexadecimal digits.

What U+XXXX means

The U+ prefix identifies Unicode code-point notation. The remaining digits are hexadecimal, so U+00E9 means hexadecimal 00E9. Code points are not the same as UTF-8 bytes: é is one code point, U+00E9, but in UTF-8 it is stored as two bytes, C3 A9.

Unicode example

The code point is stable. The byte representation depends on the encoding. UTF-8, UTF-16 and UTF-32 can store the same Unicode code point with different units and byte layouts.

Unicode vs UTF-8

Unicode and UTF-8 are related, but they are not interchangeable terms. Unicode defines characters and code points. UTF-8 is an encoding that stores those code points as bytes. UTF-8 uses one to four bytes per code point, and ASCII characters keep the same byte values they had in ASCII.

For a deeper comparison, read Unicode vs UTF-8 or try the UTF-8 Encoder and Decoder.

Unicode vs ASCII

ASCII contains 128 basic characters: English letters, digits, punctuation and control codes. Unicode includes ASCII as its first 128 code points, so A is still U+0041. UTF-8 preserves ASCII byte compatibility, which helped it become the dominant encoding for the web.

A works in ASCII and Unicode. é, Ж, 字 and 😀 require Unicode because ASCII has no way to represent them.

UTF-8, UTF-16 and UTF-32

Unicode transformation formats define how code points are represented for storage and processing. UTF-8 is compact for ASCII-heavy text and common on the web. UTF-16 is used by JavaScript, Java and many Windows APIs. UTF-32 uses a fixed 32-bit code unit for each code point, which can simplify some processing but uses more space.

One UTF-16 code unit does not always equal one character. Code points above U+FFFF, such as many emoji, use surrogate pairs in UTF-16. UTF-32 gives one code unit per code point, but that still does not guarantee one code unit per user-perceived character because grapheme clusters can contain multiple code points.

Code points, code units and grapheme clusters

When debugging text, it helps to separate bytes, code units, code points and grapheme clusters. Bytes are storage units. Code units are the units used by an encoding such as UTF-16. Code points are Unicode numbers. Grapheme clusters are what users often perceive as one character.

The family emoji 👨‍👩‍👧‍👦 may appear as one emoji, but it contains multiple code points joined by zero-width joiners. The letter é can be represented as the single code point U+00E9, or as U+0065 U+0301: the letter e followed by a combining acute accent.

Use the Unicode Sequence Analyzer to inspect these layers, and the Unicode Character Counter when code-point counts are not enough.

Combining characters and normalization

café → U+0063 U+0061 U+0066 U+00E9
café → U+0063 U+0061 U+0066 U+0065 U+0301

These two strings can look the same while their code-point sequences differ. Direct comparisons may fail, search indexes may miss matches and filenames may behave differently across systems. Unicode normalization converts equivalent representations into consistent forms.

The common normalization forms are NFC, NFD, NFKC and NFKD. NFC is often used for stored text because it prefers composed characters where possible. NFD decomposes characters into base letters and combining marks. Compatibility forms, NFKC and NFKD, also fold compatibility characters.

For details, read Unicode Normalization Explained, then try the Unicode Normalizer and Unicode Normalization Checker.

Does Unicode include emoji?

Yes. Unicode assigns code points and sequences used for emoji, but it does not control exact artwork. The same emoji can look different on Apple, Android, Windows or a web font because each platform supplies its own glyph design.

Some emoji are simple single code points. Others use variation selectors, skin-tone modifiers, regional indicators or zero-width joiners. That is why one displayed emoji can contain several code points. The Unicode Character Inspector and Unicode Sequence Analyzer are useful when emoji behavior is surprising.

Common Unicode problems

How developers work with Unicode

JavaScript

const text = "café";
const normalized = text.normalize("NFC");

for (const character of text) {
    console.log(character, character.codePointAt(0).toString(16));
}

In JavaScript, String.length counts UTF-16 code units. Iterating with for...of handles code points better than indexing, but grapheme clusters still need segmentation.

Python

import unicodedata

text = "cafe\u0301"
normalized = unicodedata.normalize("NFC", text)
print([f"U+{ord(char):04X}" for char in normalized])

Python strings are Unicode strings. The unicodedata module provides names, categories and normalization.

PHP

$text = "café";

echo mb_strlen($text, "UTF-8");

if (class_exists(Normalizer::class)) {
    $normalized = Normalizer::normalize($text, Normalizer::FORM_C);
}

In PHP, use multibyte-aware functions for Unicode text and the Intl extension when normalization is required.

Try these Unicode tools

UnicodeNow is tools-first. These pages help you inspect the exact string you are working with rather than relying on how it looks on screen.

Frequently asked questions

References

• The Unicode Standard
• Unicode Character Database
• Unicode Standard Annex #15: Unicode Normalization Forms
• Unicode Standard Annex #29: Unicode Text Segmentation
• RFC 3629: UTF-8
• Unicode glossary