--- title: "nfc_normalize" slug: "dev-database-reference-sql-functions-string-nfc-normalize" updated: 2026-10-30T18:18:56Z published: 2026-10-30T18:18:56Z canonical: "docs.vastdata.com/dev-database-reference-sql-functions-string-nfc-normalize" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # nfc_normalize *Applies to VAST Cluster 5.5.* *[Functions](/develop/docs/dev-database-reference-sql-functions-index) β€Ί [Text basics](/develop/docs/dev-database-reference-sql-functions-string-text-basics) β€Ί Unicode and URLs.* Converts a string to Unicode normalization form C (NFC), which combines a letter and a following combining accent into one precomposed character where one exists. It has no aliases. ## Syntax ``` nfc_normalize(s) ``` | Argument | Type | Description | | --- | --- | --- | | `s` | `VARCHAR` | The string to normalise. | **Returns:** `VARCHAR`. **Aliases:** none. ## Set up the examples The examples connect with the ADBC driver, as in [VAST Query Engine (ADBC)](/develop/docs/dev-database-integrations-adbc), and work in a schema called `docs_sql_fn_nfc_normalize`. Set `QE_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `VASTDB_BUCKET` ([SQL reference](/develop/docs/dev-database-reference-sql-index#example-connect-and-use-a-transaction)), and run the blocks in order in one Python session. Below each block is the output you should see. This block connects, defines the `run()` and `plan()` helpers, and creates a table of customer names loaded from two systems. The CRM wrote `José Martí` precomposed; the web shop wrote it decomposed, with combining accents. The last row has no name: ``` import os import re import adbc_driver_manager import adbc_driver_manager.dbapi as dbapi import adbc_driver_vastdb BUCKET = os.environ["VASTDB_BUCKET"] SCHEMA = f'"{BUCKET}/docs_sql_fn_nfc_normalize"' conn = dbapi.connect( driver=adbc_driver_vastdb.get_driver_path(), db_kwargs={ "vast.db.endpoint": os.environ["QE_ENDPOINT"], "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"], "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"], }, autocommit=True, ) def server_message(e): """The server's message in a driver error, without the request ID or the echoed statement.""" text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e)) found = (re.search(r'message: "(.*?)", request_id', text, re.S) or re.search(r"description: (.*)\)", text, re.S)) message = found.group(1) if found else text.strip().splitlines()[-1] message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S) message = re.split(r"\\n|\n", message)[0].replace('\\"', '"') return message.replace(BUCKET, "") def run(sql): """Run one statement. Print the rows it returns, or the error.""" try: with conn.cursor() as cur: cur.execute(sql) rows = cur.fetchall() if cur.description else [] except (adbc_driver_manager.Error, OSError) as e: print(f"{type(e).__name__}: {server_message(e)}") else: print(rows) def plan(sql): """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert.""" with conn.cursor() as cur: cur.execute("EXPLAIN " + sql) steps = cur.fetchall()[0][1] print("\n".join(line for line in steps.splitlines() if "Rows" not in line)) TABLE = f"{SCHEMA}.customers" run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}") run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, name VARCHAR, source VARCHAR)") run(f"INSERT INTO {TABLE} VALUES " "(1, 'Jos\u00e9 Mart\u00ed', 'crm'), " "(2, 'Jose\u0301 Marti\u0301', 'web'), " "(3, 'Ada Lovelace', 'web'), " "(4, NULL, 'crm')") ``` ``` [] [] [(4,)] ``` ## What it does Unicode can write many accented letters in two ways: - **precomposed:** one code point, such as `é` (U+00E9); - **decomposed:** the base letter followed by a combining accent, such as `e` and U+0301 (written `'e\u0301'` on this page). Both display as Γ©, but they're different strings: a plain `=` comparison of the two is false, and [length](/develop/docs/dev-database-reference-sql-functions-string-length) counts 1 and 2. Text from different systems, keyboards or file names can use either form. `nfc_normalize` rewrites a string in NFC, the precomposed form, so that strings that look the same compare equal: | Call | Result | | --- | --- | | `nfc_normalize('e\u0301') = 'é'` (precomposed) | `true` | | `length(nfc_normalize('e\u0301'))` | `1` | | `nfc_normalize('é') = 'é'` (already precomposed) | `true`: NFC text is left as it is | | `nfc_normalize('Cre\u0300me') = 'Crème'` | `true` | | `nfc_normalize('fi')` (the `fi` ligature) | `'fi'`: unchanged | | `nfc_normalize('\u212B') = 'Å'` (the Angstrom sign U+212B, and U+00C5) | `true`: a character with a preferred equivalent is replaced by it | It keeps the accents and the letters. It doesn't change case, doesn't remove accents (use [strip_accents](/develop/docs/dev-database-reference-sql-functions-string-strip-accents) for that), and doesn't break up compatibility characters such as the `fi` ligature into plain letters. Characters that have no accent to combine, such as emoji and `ẞ`, are returned unchanged: `'e\u0301 🇫🇷 ẞ'` comes back as `'é 🇫🇷 ẞ'`. ## NULLs and edge cases | Input | Result | | --- | --- | | `NULL` | `NULL` | | `''` (empty string) | `''` | | `'e\u0301'` (e and a combining accent) | `'é'` (one code point) | | `'é'` (precomposed) | `'é'` (unchanged) | | `'fi'` (a ligature) | `'fi'` (unchanged) | | `'e\u0301 🇫🇷 ẞ'` | `'é 🇫🇷 ẞ'` | | A number, decimal, date, boolean, `BLOB` or list | Error: `No function matches the given name and argument types` | The same cases, as the cluster returns them. They use the table and helpers from [Set up the examples](/develop/docs/dev-database-reference-sql-functions-string-nfc-normalize#set-up-the-examples): ``` run("SELECT nfc_normalize(NULL), nfc_normalize(''), nfc_normalize('e\u0301') = '\u00e9', " "length(nfc_normalize('e\u0301')), nfc_normalize('\u00e9') = '\u00e9', nfc_normalize('\ufb01')") run("SELECT 'e\u0301' = '\u00e9', nfc_normalize('e\u0301 πŸ‡«πŸ‡· ẞ') = '\u00e9 πŸ‡«πŸ‡· ẞ'") run("SELECT nfc_normalize(123)") ``` ``` [(None, '', True, 1, True, 'fi')] [(False, True)] InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(INTEGER_LITERAL)'. You might need to add explicit type casts. ``` ## Examples The two spellings of `José Martí` have different lengths until they're normalised: ``` run(f"SELECT id, source, length(name), length(nfc_normalize(name)) FROM {TABLE} ORDER BY id LIMIT 10") ``` ``` [(1, 'crm', 10, 10), (2, 'web', 12, 10), (3, 'web', 12, 12), (4, 'crm', None, None)] ``` A plain comparison finds only the precomposed row. Comparing `nfc_normalize(name)` finds both: ``` run(f"SELECT id, source FROM {TABLE} WHERE name = 'Jos\u00e9 Mart\u00ed' ORDER BY id LIMIT 10") run(f"SELECT id, source FROM {TABLE} WHERE nfc_normalize(name) = 'Jos\u00e9 Mart\u00ed' ORDER BY id LIMIT 10") ``` ``` [(1, 'crm')] [(1, 'crm'), (2, 'web')] ``` ## How it runs in VAST The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (`VAST_SCAN`). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so `nfc_normalize()` is always computed by the Query Engine: - **In the `SELECT` list,** the scan reads the column and the Query Engine computes `nfc_normalize()` for each row it returns. - **In `WHERE`,** the scan sends the column for every row, and a `FILTER` step in the Query Engine computes `nfc_normalize()` and keeps the rows that match. No rows are skipped in storage. The plan of the `WHERE` example shows the `FILTER` step above the scan: ``` plan(f"SELECT id FROM {TABLE} WHERE nfc_normalize(name) = 'Jos\u00e9 Mart\u00ed'") ``` ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ PROJECTION β”‚ β”‚ ──────────────────── β”‚ β”‚ #1 β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ FILTER β”‚ β”‚ ──────────────────── β”‚ β”‚ (nfc_normalize(name) = β”‚ β”‚ 'JosΓ© MartΓ­') β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ VAST_SCAN β”‚ β”‚ ──────────────────── β”‚ β”‚ Function: VAST_SCAN β”‚ β”‚ β”‚ β”‚ Projections: β”‚ β”‚ name β”‚ β”‚ id β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Make it faster - **Normalise when you write, not when you read.** Store the NFC form of the text: normalise it in the application that loads it. Then a plain comparison finds every spelling, and a comparison of the raw column with a constant is pushed down to VAST DataBase, so rows that don't match never leave storage. The plan for `name = 'José Martí'` has no `FILTER` step: ``` plan(f"SELECT id FROM {TABLE} WHERE name = 'Jos\u00e9 Mart\u00ed'") ``` ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ VAST_SCAN β”‚ β”‚ ──────────────────── β”‚ β”‚ Function: VAST_SCAN β”‚ β”‚ Projections: id β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` - **Write the search value in NFC.** If the column is stored in NFC, write the constant you compare it with in NFC too (precomposed), in your application, or normalise the constant in SQL: in `name = nfc_normalize('Jose\u0301 Marti\u0301')`, `nfc_normalize()` of a constant is computed once, and the comparison of the raw column with the result is pushed down to VAST DataBase. - **Combine it with a pushed-down condition.** In `source = 'web' AND nfc_normalize(name) = 'José Martí'`, `source = 'web'` is still evaluated in VAST DataBase, so `nfc_normalize()` runs only on the rows that pass it. ## Limitations and errors - **Strings only.** A number, decimal, date, boolean, `BLOB` or list fails with `No function matches the given name and argument types`. - **NFC only.** It composes; it doesn't change case, remove accents, or replace compatibility characters such as ligatures with plain letters. To ignore case as well, compare [lower](/develop/docs/dev-database-reference-sql-functions-string-lower) of the normalised strings; to ignore accents, use [strip_accents](/develop/docs/dev-database-reference-sql-functions-string-strip-accents). - **No other normalization forms.** `nfd_normalize` and `nfkc_normalize` fail with `Scalar Function with name ... does not exist!`. ``` run("SELECT nfc_normalize(123)") run("SELECT nfc_normalize('\\xAA'::BLOB)") run("SELECT nfc_normalize(['a', 'b'])") ``` ``` InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(INTEGER_LITERAL)'. You might need to add explicit type casts. InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(BLOB)'. You might need to add explicit type casts. InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(VARCHAR[])'. You might need to add explicit type casts. ``` ## Similar functions | Function | What it does to `'e\u0301'` (e and a combining accent) | Result | | --- | --- | --- | | `nfc_normalize` | composes it into one code point | `'é'`; `length` of the result is `1` | | [strip_accents](/develop/docs/dev-database-reference-sql-functions-string-strip-accents) | removes the accent | `'e'` | | [length_grapheme](/develop/docs/dev-database-reference-sql-functions-string-length-grapheme) | counts what a reader sees, without changing the string | `1` | | [length](/develop/docs/dev-database-reference-sql-functions-string-length) | counts code points, without changing the string | `2` | | [COLLATE noaccent](/develop/docs/dev-database-reference-sql-functions-string-collate) | compares ignoring accents, without changing the string | `'é' COLLATE noaccent = 'e'` is `true` | On `'Crème'`, `nfc_normalize` turns `'Cre\u0300me'` (`e` followed by the combining grave accent U+0300) into the precomposed `'Crème'`; `strip_accents` of the precomposed `'Crème'` gives `'Creme'`; and `COLLATE noaccent` still compares case: `'Crème' COLLATE noaccent = 'creme'` is `false`. ``` run("SELECT nfc_normalize('e\u0301') = '\u00e9', length(nfc_normalize('e\u0301')), strip_accents('e\u0301'), " "length_grapheme('e\u0301'), length('e\u0301')") run("SELECT '\u00e9' COLLATE noaccent = 'e'") run("SELECT strip_accents('Cr\u00e8me'), nfc_normalize('Cre\u0300me') = 'Cr\u00e8me', " "'Cr\u00e8me' COLLATE noaccent = 'creme'") ``` ``` [(True, 1, 'e', 1, 2)] [(True,)] [('Creme', True, False)] ``` Clean up: ``` run(f"DROP TABLE IF EXISTS {TABLE}") run(f"DROP SCHEMA IF EXISTS {SCHEMA}") conn.close() ``` ``` [] [] ``` ## See also - [Text basics](/develop/docs/dev-database-reference-sql-functions-string-text-basics): all the text functions. - [Where filters run](/develop/docs/dev-database-reference-sql-select#where-filters-run) and [pushdowns](/develop/docs/dev-database-reference-pushdowns). - [EXPLAIN](/develop/docs/dev-database-reference-sql-explain): how to read a plan. - [strip_accents](/develop/docs/dev-database-reference-sql-functions-string-strip-accents), [length_grapheme](/develop/docs/dev-database-reference-sql-functions-string-length-grapheme) and [COLLATE nocase and noaccent](/develop/docs/dev-database-reference-sql-functions-string-collate).