bit_position

Applies to VAST Cluster 5.5.

Functions › Binary, hash and UUID functions › Bit strings. Returns the position of the first occurrence of one bit string in another, counting from 1 at the left, or 0 if it doesn't occur: bit_position('11'::BIT, '0011'::BIT) is 3.

Syntax

bit_position(substring, bits)

Returns: INTEGER.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_binary_hash_uuid_bit_position. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of five feature-flag sets, stored as VARCHAR strings of 0s and 1s. The third row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_binary_hash_uuid_bit_position"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.flags"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, bits VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, '1011'), "
    "(2, '0'), "
    "(3, NULL), "
    "(4, '10000000'), "
    "(5, '0110')")

[]
[]
[(5,)]

What it does

bit_position scans bits from the left and returns the position where substring first starts. The first bit is position 1. It returns 0 when substring doesn't occur, including when it's longer than bits.

With substring set to '1'::BIT, it gives the position of the first bit that is set, so bit_position('1'::BIT, bits) - 1 is the number of leading 0s. Note that get_bit and set_bit count from 0, not 1.

String literals of 0s and 1s are cast to BIT, so bit_position('1', '0011') works. A VARCHAR column must be cast: CAST(bits AS BIT).

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT bit_position(NULL, NULL), bit_position('1'::BIT, NULL::BIT), bit_position(NULL::BIT, '1'::BIT)")
run("SELECT bit_position('111'::BIT, '0011'::BIT), bit_position('0'::BIT, '1111'::BIT), "
    "bit_position('0011'::BIT, '0011'::BIT), bit_position('00111'::BIT, '0011'::BIT), "
    "bit_position('10'::BIT, '101010'::BIT), bit_position('1', '0011')")
run("SELECT bit_position('abc', '0011'::BIT)")

[(None, None, None)]
[(0, 0, 1, 0, 1, 3)]
OSError: Invalid character encountered in string -> bit conversion: 'a'

Examples

The position of the first flag that is set in each row. Row 2 has none, so its result is 0:

run(f"SELECT id, bits, bit_position('1'::BIT, CAST(bits AS BIT)) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, '1011', 1), (2, '0', 0), (3, None, None), (4, '10000000', 1), (5, '0110', 2)]

Keep the rows that contain the pattern 11:

run(f"SELECT id, bits FROM {TABLE} WHERE bit_position('11'::BIT, CAST(bits AS BIT)) > 0 ORDER BY id LIMIT 10")

[(1, '1011'), (5, '0110')]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so bit_position() is always computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes bit_position() for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine computes bit_position() and keeps the rows that match. No rows are skipped in storage.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE bit_position('11'::BIT, CAST(bits AS BIT)) > 0")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│   (bit_position(11, CAST  │
│    (bits AS BIT)) > 0)    │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│            bits           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• A text search isn't faster. Searching the VARCHAR column with a string function or LIKE runs in a FILTER step too: the plan for bits LIKE '1%' has one.

plan(f"SELECT id FROM {TABLE} WHERE bits LIKE '1%'")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│     prefix(bits, '1')     │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│            bits           │
│             id            │
│                           │
└───────────────────────────┘

• Store what you filter on. If you often look for one pattern, compute the result when you write the rows and keep it in its own column: a comparison of a raw column with a constant is pushed down to VAST DataBase.
• Combine it with a pushed-down condition. In id > 100 AND bit_position('11'::BIT, CAST(bits AS BIT)) > 0, id > 100 is still evaluated in VAST DataBase, so bit_position() runs only on the rows that pass it.

Limitations and errors

• Both arguments must be bit strings. A number, BLOB, date, boolean, list or VARCHAR column fails with No function matches the given name and argument types, for example 'bit_position(BIT, VARCHAR)'. The error lists the one form bit_position accepts: bit_position(BIT, BIT). Cast a VARCHAR column with CAST(col AS BIT).

run(f"SELECT bit_position('1'::BIT, bits) FROM {TABLE} LIMIT 1")

InternalError: Binder Error: No function matches the given name and argument types 'bit_position(BIT, VARCHAR)'. You might need to add explicit type casts.

• Only the first occurrence. To test whether a pattern occurs, compare the result with 0.

Similar functions

run("SELECT bit_position('1'::BIT, '0101'::BIT), get_bit('0101'::BIT, 1), bit_count('0101'::BIT)")

[(2, 1, 2)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Binary, hash and UUID functions: all the bit string functions.
• BIT values and casts.
• Where filters run and pushdowns.