UnicodeNow Data Sources

Unicode Character Database

The project downloads the following files directly from unicode.org (scripts/download_unicode_data.py) into data/ucd/. Only files actually present and read by the application are listed.

ScriptExtensions.txt, DerivedCoreProperties.txt, PropList.txt, EastAsianWidth.txt, NameAliases.txt, BidiMirroring.txt, DerivedBidiClass.txt, GraphemeBreakProperty.txt, WordBreakProperty.txt, SentenceBreakProperty.txt and NormalizationTest.txt are not currently downloaded or used; features that would depend on them (Script Extensions, East Asian width, a dedicated grapheme/word break table, mirroring partner pairs) are Not bundled rather than approximated.

Python Unicode metadata

The deployed server runs Python 3.13.14, whose bundled unicodedata module reports unicodedata.unidata_version as 15.1.0. This is Active for the properties it supplies and is the source of truth for character identity across server-side code.

Functions used: unicodedata.name, unicodedata.category, unicodedata.combining, unicodedata.bidirectional, unicodedata.normalize.

Python unicodedata does not by itself supply script, block, Unicode age, full emoji-sequence data, complete named HTML entities, or full Unicode case-fold tables; those come from the separately bundled files described in the sections below. unicodedata.east_asian_width and unicodedata.mirrored are available in the standard library but are not currently called by this project, since East Asian width is not exposed by any tool.

Scripts and Script Extensions

Script values come from the bundled Scripts.txt (Active), parsed into range data and used directly by the affected tools without going through Python unicodedata. Script_Extensions data (ScriptExtensions.txt) is Not bundled, so tools report only the primary Script property.

Script and Script_Extensions are different properties: Script assigns exactly one value per code point, while Script_Extensions can list several scripts a shared code point is used with. Common and Inherited are themselves valid Script values, used respectively for code points shared across scripts (digits, punctuation, most emoji) and for combining marks that take their script context from a preceding base character. Where a tool associates a Common or Inherited code point with a neighboring substantive script for readability, that association is derived, documented behavior, not the raw Unicode property; the raw Script value remains available alongside it.

Used by: Unicode Character Lookup, Unicode Sequence Analyzer and Mixed Script Detector. The Unicode Character Inspector uses a separate, coarser heuristic script classifier rather than this bundled file, and its script output should be treated as approximate.

Blocks and planes

Block ranges come from the bundled Blocks.txt (Active). A block is a contiguous range of code points reserved for a purpose; it is not the same property as script, and a single block can contain characters from several different scripts. Unassigned code points can still fall inside a named block. Plane labels (Basic Multilingual Plane, Supplementary Multilingual Plane and so on) are derived directly from fixed code-point range boundaries defined by the Unicode Standard, not read from a separate file.

Used by: Unicode Character Lookup and Unicode Sequence Analyzer.

Emoji data

The bundled emoji-data.txt declares Emoji 17.0 (UTS #51). emoji-sequences.txt, emoji-zwj-sequences.txt, emoji-variation-sequences.txt and emoji-test.txt are Not bundled.

The Unicode Sequence Analyzer detects structural indicators such as ZWJ sequences, regional-indicator (flag) pairs, emoji modifiers, variation selectors, keycap sequences and tag sequences using code-point range checks and grapheme segmentation, but this does not amount to a complete classification against the full registered emoji-sequence files. It does not claim complete emoji-sequence classification.

Grapheme and word segmentation

Grapheme-cluster segmentation prefers the browser's built-in Intl.Segmenter with granularity: "grapheme" (Active where supported). Word segmentation similarly prefers Intl.Segmenter with granularity: "word", falling back to a regular-expression approximation of word-like runs where unavailable. There is no server-side segmentation implementation and no bundled UCD grapheme/word break-property file backing a custom fallback; the fallback is a plain code-point iteration.

Array.from(text) counts code points, not grapheme clusters, and this site never describes it as grapheme segmentation. Segmentation uses the browser's default locale; it is not customized per language. Emoji ZWJ sequences, combining marks and regional-indicator pairs are grouped into one grapheme cluster only when Intl.Segmenter is available; the code-point fallback counts each constituent code point separately instead. Segmentation results can differ between browsers, and between browser versions, because they depend on the Unicode version each browser's Intl.Segmenter implementation bundles; this project does not test or guarantee identical segmentation across browsers.

Bidirectional and combining properties

Bidirectional class and canonical combining class come from Python's unicodedata module server-side, and from the bundled UnicodeData.txt fields when read directly in generated browser data (Active for both). The mirrored (Y/N) flag is read from UnicodeData.txt; specific mirroring partner pairs would require BidiMirroring.txt, which is Not bundled, so only the boolean flag is available.

UnicodeNow reports raw bidi-class properties and groups code points into directionality runs based on those classes. It does not implement the full Unicode Bidirectional Algorithm and does not claim to reproduce exact visual rendering order; a browser's actual layout can still differ. Bidirectional control characters and directional isolates are never rendered raw in diagnostic tables: they are shown as a visible bracketed label and isolated (using <bdi> or equivalent) so they cannot reorder surrounding page content.

HTML entities and encoding references

Named HTML character references come from Python's built-in html.entities module (WHATWG-aligned named character references), used server-side where HTML entity output is offered. Named entities are an HTML feature, not a Unicode property, and not every Unicode character has one; numeric decimal and hexadecimal references are shown instead, and those are valid for any Unicode scalar value.

The following are calculated directly from the input text at request time, not imported from a data file: UTF-8 bytes, UTF-16 code units, UTF-32 values, HTML numeric (decimal and hexadecimal) references, JavaScript escapes, JSON escapes, CSS escapes and URL encoding.

Browser APIs

Browser APIs are runtime behavior, not a bundled dataset shipped by this project; the same input can produce different segmentation or normalization results on different browsers because each browser supplies its own Unicode version.

Source-to-tool mapping

Data import and build process

1. Source files are fetched from unicode.org by scripts/download_unicode_data.py into data/ucd/, which is committed to the repository.
2. Each downloaded file's own header comment records its Unicode version and date, which this project reads at load time rather than duplicating manually.
3. Server-side loaders (app/unicode_data/blocks.py, scripts.py, age.py, confusables.py) parse the relevant text file into range tables on first use and cache the result in memory.
4. A separate build script, scripts/generate_unicode_property_data.py, parses UnicodeData.txt and Blocks.txt into a compact, range-compressed JSON bundle committed as app/static/js/unicode-property-data.js, so the Unicode Sequence Analyzer can look up category, combining class, bidi class, mirrored status, block and (for the Basic Multilingual Plane and emoji range) name entirely in the browser.
5. A similar generated bundle, app/static/js/unicode-script-data.js, provides script ranges to browser-local tools; unlike the property-data bundle, its generator script is not currently committed to the repository.
6. Generated JavaScript bundles are committed to the repository, the same as the source data files; there is no build step that regenerates them automatically during deployment.
7. A separate, optional script pair (scripts/import_unicode_data.py and scripts/seed_database.py) can populate a Postgres unicode_characters table from UnicodeData.txt, but neither script runs automatically, and the live Character Lookup route does not query that table.
8. There is no automated regression suite specifically for the data-generation scripts themselves; correctness is checked through the application-level tests described below.
9. There is no dataset build manifest file; this page reads the same version metadata (Python's unicodedata.unidata_version and each loader's parsed version header) that the live tools use, so the page and the tools cannot drift apart from each other.
10. If a source file is missing, the affected loader falls back to a small hardcoded range table (blocks) or reports "Unknown"/empty data (scripts, age) rather than failing the page.

No infrastructure credentials, deployment secrets or private filesystem paths are involved in this process beyond the repository-relative data/ucd/ directory referenced above.

Versioning and updates

Unicode data is updated manually: a maintainer re-runs scripts/download_unicode_data.py to fetch current files, then re-runs the generation scripts described above and commits the results. There is no scheduled or automatic update process tied to new Unicode releases. Because the Python runtime version is pinned by the deployment environment, unicodedata.unidata_version only changes when the deployed Python version changes, independently of when the bundled data files are refreshed. Updates are tested using this project's existing automated test suite rather than a dedicated dataset-diffing process. Past results remain reproducible only to the extent that the same Python version and the same committed data files are used; this page does not maintain a public changelog of prior dataset versions.

Validation and consistency checks

Automated tests in this project's test suite check: Unicode scalar-value validation (rejecting values above U+10FFFF), rejection of UTF-16 surrogate code points (U+D800–U+DFFF) as standalone scalar values, non-overlapping and ordered code-point ranges in the generated block/script/property tables, decomposed-versus-precomposed normalization behavior against hand-verified fixtures, grapheme-cluster counts for combining marks, flags, emoji modifiers and ZWJ sequences against hand-verified fixtures, safe rendering of bidi controls and invisible characters, and that browser-local tools make no network requests during analysis. Where both a server-side and a client-side implementation of the same operation exist, tests check them against known fixture values rather than against each other directly; this project does not currently run automated parity tests comparing every client implementation to every server implementation. Production URLs in metadata and JSON-LD are checked by tests to ensure no local development URL is present.

Licensing and attribution

Unicode data files distributed by unicode.org, including those bundled in this project, are provided under Unicode, Inc.'s terms of use. UnicodeNow is an independent website and is not affiliated with or endorsed by Unicode, Inc. Third-party software libraries used by this project (including the web framework, HTTP client and other dependencies) carry their own separate licenses. Browser APIs such as Intl.Segmenter are platform implementations provided by browser vendors, not data shipped by this project. Generated files such as unicode-property-data.js and unicode-script-data.js are derived directly from the official Unicode data files described above.

Known limitations

• The Python Unicode version (15.1.0) can lag behind the latest published Unicode release.
• Browser segmentation results can differ from this site's bundled data, since each browser bundles its own Unicode version.
• Glyph rendering always depends on the visitor's installed fonts and operating system.
• Emoji appearance is a font and platform concern; it is not supplied by any Unicode data file this project bundles.
• Script resolution for combined "resolved script" views uses a documented derived heuristic, not a raw Unicode property.
• Private-use character meanings are application-specific and cannot be resolved by this project.
• Unassigned status depends on the specific bundled or runtime Unicode version described above.
• Named HTML entities cover only part of Unicode; most characters only have numeric references.
• Full Unicode case folding requires CaseFolding.txt data; while that file is bundled, it is not currently wired into a live tool, so lowercase comparisons use simple lowercasing instead.
• A valid, correctly identified code point may still fail to render if no installed font supports it.
• Browser APIs may use a different Unicode version than the data bundled with this site.

Frequently asked questions

Related resources

Methodology · Privacy · About · Contact · What Is Unicode? · Code Points vs Code Units · What Is a Grapheme Cluster? · What Are Invisible Unicode Characters? · HTML Entities vs Unicode Characters

Representative tools referenced throughout this page: