bit_count

Applies to VAST Cluster 5.5.

Functions › Binary, hash and UUID functions › Bit strings. Returns the number of bits set to 1 in a BIT value, or in the two's-complement form of an integer: bit_count('101101'::BIT) is 4.

Syntax

bit_count(x)

Returns: BIGINT for a BIT value; TINYINT for an integer.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_binary_hash_uuid_bit_count. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of five feature-flag sets. Each is stored twice: as a VARCHAR of 0s and 1s, and as an INTEGER permission mask. The third row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_binary_hash_uuid_bit_count"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.flags"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, bits VARCHAR, perms INTEGER)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, '1011', 11), "
    "(2, '0', 0), "
    "(3, NULL, NULL), "
    "(4, '10000000', 128), "
    "(5, '0110', -1)")

[]
[]
[(5,)]

What it does

bit_count counts the 1s:

The width matters for negative numbers: -1 has 8 bits set as a TINYINT, 16 as a SMALLINT, 32 as an INTEGER and 64 as a BIGINT. A whole-number literal such as -1 is an INTEGER.

A VARCHAR column must be cast to BIT first: bit_count(CAST(bits AS BIT)).

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT bit_count(NULL), bit_count('0'::BIT), bit_count(CAST(repeat('1', 1000) AS BIT)), bit_count(123), "
    "bit_count(-1::TINYINT), bit_count(-1::BIGINT)")
run("SELECT bit_count(-128::TINYINT)")
run("SELECT bit_count('5')")
run("SELECT bit_count(1.5)")

[(None, 0, 1000, 6, 8, 64)]
OSError: Type INT32 with value 128 can't be cast because the value is out of range for the destination type INT8
InternalError: Binder Error: Could not choose a best candidate function for the function call "bit_count(STRING_LITERAL)". In order to select one, please add explicit type casts.
InternalError: Binder Error: No function matches the given name and argument types 'bit_count(DECIMAL(2,1))'. You might need to add explicit type casts.

Examples

The number of flags set in each row, from the bit string and from the integer mask. -1 as an INTEGER has all 32 bits set:

run(f"SELECT id, bits, bit_count(CAST(bits AS BIT)), perms, bit_count(perms) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, '1011', 3, 11, 3), (2, '0', 0, 0, 0), (3, None, None, None, None), (4, '10000000', 1, 128, 1), (5, '0110', 2, -1, 32)]

Keep the rows with more than two flags set:

run(f"SELECT id, bits FROM {TABLE} WHERE bit_count(CAST(bits AS BIT)) > 2 ORDER BY id LIMIT 10")

[(1, '1011')]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so bit_count() is always computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes bit_count() for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine computes bit_count() and keeps the rows that match. No rows are skipped in storage.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE bit_count(CAST(bits AS BIT)) > 2")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│  (bit_count(CAST(bits AS  │
│         BIT)) > 2)        │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│            bits           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Store the count if you filter on it often. Compute it when you write the rows and keep it in its own column: flag_count > 2 is pushed down to VAST DataBase, bit_count(CAST(bits AS BIT)) > 2 isn't. The plan for a raw column compared with a constant has no FILTER step:

plan(f"SELECT id FROM {TABLE} WHERE perms > 2")

┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘

• Combine it with a pushed-down condition. In id > 100 AND bit_count(perms) > 2, id > 100 is still evaluated in VAST DataBase, so bit_count() runs only on the rows that pass it.

Limitations and errors

• HUGEINT counts above 127 wrap. The result for an integer is a TINYINT, so a HUGEINT with 128 bits set returns -128: bit_count(-1::HUGEINT) is -128, not 128. For a HUGEINT that can be negative, count the bits of its cast to BIT, which returns a BIGINT.

run("SELECT bit_count(-1::HUGEINT), bit_count(CAST(-1::HUGEINT AS BIT))")

[(-128, 128)]

• Not for UHUGEINT or VARINT. They fail with No function matches the given name and argument types.
• Not for strings, decimals or floating-point numbers. A VARCHAR column, a DECIMAL, a DOUBLE, a BOOLEAN or a date fails with No function matches the given name and argument types. The error lists the types bit_count accepts. Cast a VARCHAR of 0s and 1s to BIT first.
• Negative literals of a small type need parentheses. -128::TINYINT casts 128 and then negates it, and 128 doesn't fit in a TINYINT. Write (-128)::TINYINT.

Similar functions

run("SELECT bit_count('101101'::BIT), length('101101'::BIT), bit_length('101101'::BIT), "
    "octet_length('101101'::BIT), get_bit('101101'::BIT, 0), bit_position('1'::BIT, '101101'::BIT)")

[(4, 6, 6, 1, 1, 1)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Binary, hash and UUID functions: all the bit string functions.
• BIT values and casts.
• Where filters run and pushdowns.