nfc_normalize

Applies to VAST Cluster 5.5.

Functions › Text basics › Unicode and URLs. Converts a string to Unicode normalization form C (NFC), which combines a letter and a following combining accent into one precomposed character where one exists. It has no aliases.

Syntax

nfc_normalize(s)

Returns: VARCHAR.

Aliases: none.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_nfc_normalize. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of customer names loaded from two systems. The CRM wrote José Martí precomposed; the web shop wrote it decomposed, with combining accents. The last row has no name:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_nfc_normalize"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.customers"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, name VARCHAR, source VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, 'Jos\u00e9 Mart\u00ed', 'crm'), "
    "(2, 'Jose\u0301 Marti\u0301', 'web'), "
    "(3, 'Ada Lovelace', 'web'), "
    "(4, NULL, 'crm')")

[]
[]
[(4,)]

What it does

Unicode can write many accented letters in two ways:

• precomposed: one code point, such as é (U+00E9);
• decomposed: the base letter followed by a combining accent, such as e and U+0301 (written 'e\u0301' on this page).

Both display as é, but they're different strings: a plain = comparison of the two is false, and length counts 1 and 2. Text from different systems, keyboards or file names can use either form. nfc_normalize rewrites a string in NFC, the precomposed form, so that strings that look the same compare equal:

It keeps the accents and the letters. It doesn't change case, doesn't remove accents (use strip_accents for that), and doesn't break up compatibility characters such as the fi ligature into plain letters. Characters that have no accent to combine, such as emoji and ẞ, are returned unchanged: 'e\u0301 🇫🇷 ẞ' comes back as 'é 🇫🇷 ẞ'.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT nfc_normalize(NULL), nfc_normalize(''), nfc_normalize('e\u0301') = '\u00e9', "
    "length(nfc_normalize('e\u0301')), nfc_normalize('\u00e9') = '\u00e9', nfc_normalize('\ufb01')")
run("SELECT 'e\u0301' = '\u00e9', nfc_normalize('e\u0301 🇫🇷 ẞ') = '\u00e9 🇫🇷 ẞ'")
run("SELECT nfc_normalize(123)")

[(None, '', True, 1, True, 'fi')]
[(False, True)]
InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(INTEGER_LITERAL)'. You might need to add explicit type casts.

Examples

The two spellings of José Martí have different lengths until they're normalised:

run(f"SELECT id, source, length(name), length(nfc_normalize(name)) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 'crm', 10, 10), (2, 'web', 12, 10), (3, 'web', 12, 12), (4, 'crm', None, None)]

A plain comparison finds only the precomposed row. Comparing nfc_normalize(name) finds both:

run(f"SELECT id, source FROM {TABLE} WHERE name = 'Jos\u00e9 Mart\u00ed' ORDER BY id LIMIT 10")
run(f"SELECT id, source FROM {TABLE} WHERE nfc_normalize(name) = 'Jos\u00e9 Mart\u00ed' ORDER BY id LIMIT 10")

[(1, 'crm')]
[(1, 'crm'), (2, 'web')]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so nfc_normalize() is always computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes nfc_normalize() for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine computes nfc_normalize() and keeps the rows that match. No rows are skipped in storage.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE nfc_normalize(name) = 'Jos\u00e9 Mart\u00ed'")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│  (nfc_normalize(name) =   │
│       'José Martí')       │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│            name           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Normalise when you write, not when you read. Store the NFC form of the text: normalise it in the application that loads it. Then a plain comparison finds every spelling, and a comparison of the raw column with a constant is pushed down to VAST DataBase, so rows that don't match never leave storage. The plan for name = 'José Martí' has no FILTER step:

plan(f"SELECT id FROM {TABLE} WHERE name = 'Jos\u00e9 Mart\u00ed'")

┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘

• Write the search value in NFC. If the column is stored in NFC, write the constant you compare it with in NFC too (precomposed), in your application, or normalise the constant in SQL: in name = nfc_normalize('Jose\u0301 Marti\u0301'), nfc_normalize() of a constant is computed once, and the comparison of the raw column with the result is pushed down to VAST DataBase.
• Combine it with a pushed-down condition. In source = 'web' AND nfc_normalize(name) = 'José Martí', source = 'web' is still evaluated in VAST DataBase, so nfc_normalize() runs only on the rows that pass it.

Limitations and errors

• Strings only. A number, decimal, date, boolean, BLOB or list fails with No function matches the given name and argument types.
• NFC only. It composes; it doesn't change case, remove accents, or replace compatibility characters such as ligatures with plain letters. To ignore case as well, compare lower of the normalised strings; to ignore accents, use strip_accents.
• No other normalization forms. nfd_normalize and nfkc_normalize fail with Scalar Function with name ... does not exist!.

run("SELECT nfc_normalize(123)")
run("SELECT nfc_normalize('\\xAA'::BLOB)")
run("SELECT nfc_normalize(['a', 'b'])")

InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(INTEGER_LITERAL)'. You might need to add explicit type casts.
InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(BLOB)'. You might need to add explicit type casts.
InternalError: Binder Error: No function matches the given name and argument types 'nfc_normalize(VARCHAR[])'. You might need to add explicit type casts.

Similar functions

On 'Crème', nfc_normalize turns 'Cre\u0300me' (e followed by the combining grave accent U+0300) into the precomposed 'Crème'; strip_accents of the precomposed 'Crème' gives 'Creme'; and COLLATE noaccent still compares case: 'Crème' COLLATE noaccent = 'creme' is false.

run("SELECT nfc_normalize('e\u0301') = '\u00e9', length(nfc_normalize('e\u0301')), strip_accents('e\u0301'), "
    "length_grapheme('e\u0301'), length('e\u0301')")
run("SELECT '\u00e9' COLLATE noaccent = 'e'")
run("SELECT strip_accents('Cr\u00e8me'), nfc_normalize('Cre\u0300me') = 'Cr\u00e8me', "
    "'Cr\u00e8me' COLLATE noaccent = 'creme'")

[(True, 1, 'e', 1, 2)]
[(True,)]
[('Creme', True, False)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Text basics: all the text functions.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.
• strip_accents, length_grapheme and COLLATE nocase and noaccent.