md5_number_lower

Applies to VAST Cluster 5.5.

Functions › Binary, hash and UUID functions › Hashes. Returns the last 8 bytes of the MD5 digest of a string or of a BLOB, as an unsigned 64-bit UBIGINT. md5_number_upper returns the other 8 bytes.

Syntax

md5_number_lower(x)

Returns: UBIGINT, an unsigned 64-bit integer (0 or more).

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_binary_hash_uuid_md5_number_lower. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four documents, each with a name and its first bytes. The last row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_binary_hash_uuid_md5_number_lower"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.documents"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, name VARCHAR, data BLOB)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, 'hello', '\\x25\\x50\\x44\\x46'::BLOB), "
    "(2, '', ''::BLOB), "
    "(3, 'a', '\\x61'::BLOB), "
    "(4, NULL, NULL)")

[]
[]
[(4,)]

What it does

md5_number_lower computes the 16-byte MD5 digest, as md5 does, takes its last 8 bytes, and reads them as an unsigned 64-bit integer, least significant byte first (little-endian):

The result is never negative. It's a 64-bit value that fits a UBIGINT column, so it's a compact key or checksum taken from the digest. A VARCHAR is hashed as its UTF-8 bytes, so md5_number_lower('hello') and md5_number_lower('hello'::BLOB) are equal.

NULLs and edge cases

The same cases, as the cluster returns them. They use the helpers from Set up the examples:

run("SELECT md5_number_lower(NULL), md5_number_lower(''), md5_number_lower('hello') = md5_number_lower('hello'::BLOB)")
run("SELECT md5_number_lower('a') < 0, typeof(md5_number_lower('a'))")
run("SELECT md5('abc'), hex(md5_number_lower('abc')), md5('hello'), hex(md5_number_lower('hello'))")
run("SELECT md5_number_lower(123)")

[(None, 9098107892288553193, True)]
[(False, 'UBIGINT')]
[('900150983cd24fb0d6963f7d28e17f72', '727FE1287D3F96D6', '5d41402abc4b2a76b9719d911017c592', '92C51710919D71B9')]
InternalError: Binder Error: No function matches the given name and argument types 'md5_number(INTEGER_LITERAL)'. You might need to add explicit type casts.

Examples

The last 8 bytes of the digest of each document's name and of its bytes:

run(f"SELECT id, md5_number_lower(name), md5_number_lower(data) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 10575884659879408057, 3474428166648233324), (2, 9098107892288553193, 9098107892288553193), (3, 7000413967451013937, 7000413967451013937), (4, None, None)]

Spread the documents over 4 buckets by the digest of their name:

run(f"SELECT id, name, md5_number_lower(name) % 4 FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 'hello', 1), (2, '', 1), (3, 'a', 1), (4, None, None)]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so md5_number_lower() of a column is always computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes the value for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine hashes each value and keeps the rows that match. No rows are skipped in storage.

The plan of a filter on md5_number_lower() shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE md5_number_lower(name) IS NOT NULL")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│   (CAST(CAST(array_slice  │
│ (CAST(CAST(md5_number(name│
│ ) AS BIT) AS VARCHAR), [],│
│   64) AS BIT) AS UBIGINT) │
│        IS NOT NULL)       │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│            name           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Store the value if you filter or look rows up by it. Compute it when you write the rows and keep it in a UBIGINT column. A comparison of that column with a constant is evaluated in VAST DataBase; md5_number_lower(name) = ... hashes every row in the Query Engine.

Limitations and errors

• Strings and bytes only. Other types fail with No function matches the given name and argument types. The error names md5_number rather than md5_number_lower, and lists md5_number(VARCHAR) and md5_number(BLOB). Cast a number or a UUID to VARCHAR first.
• Not the bottom half of md5_number. Despite the name, md5_number_upper(x) * 2^64 + md5_number_lower(x) is not md5_number(x), and the low 64 bits of md5_number(x) are not md5_number_lower(x). Use the bytes described in What it does.
• Half of the digest. 64 bits are enough for bucketing and checksums; for a key that must not collide across many rows, keep the full digest with md5 or sha256.

run("SELECT md5_number_lower(DATE '2026-10-16')")
run("SELECT (md5_number('abc') & 18446744073709551615)::UBIGINT = md5_number_lower('abc')")

InternalError: Binder Error: No function matches the given name and argument types 'md5_number(DATE)'. You might need to add explicit type casts.
[(False,)]

Similar functions

run("SELECT md5('abc'), md5_number('abc'), md5_number_upper('abc'), md5_number_lower('abc')")
run("SELECT hex(md5_number('abc')), hex(md5_number_upper('abc')), hex(md5_number_lower('abc'))")

[('900150983cd24fb0d6963f7d28e17f72', Decimal('152195979970564155685860391459828531600'), 12704604231530709392, 8250560606382298838)]
[('727FE1287D3F96D6B04FD23C98500190', 'B04FD23C98500190', '727FE1287D3F96D6')]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Binary, hash and UUID functions: all the functions in this family.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.