& (bitwise AND)

Applies to VAST Cluster 5.5.

Functions › Operators and predicates › Bitwise operators. a & b returns an integer whose bits are set where they're set in both a and b. Use it to test or keep flags in a bit mask: perms & 2 keeps only the bit with value 2.

Syntax

a & b

Returns: the integer type of the arguments. When they differ, the result is the wider type: INTEGER & BIGINT is a BIGINT, and a signed and an unsigned type of the same size, such as TINYINT & UTINYINT, give a BIGINT.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_operators_bitwise_and. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four users with a permission mask: 1 is read, 2 is write and 4 is admin. The last row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_operators_bitwise_and"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.users"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, name VARCHAR, perms INTEGER)")
run(f"INSERT INTO {TABLE} VALUES (1, 'ada', 7), (2, 'grace', 5), (3, 'alan', 1), (4, NULL, NULL)")

[]
[]
[(4,)]

What it does

& compares the two values bit by bit and sets a bit in the result only where both have it set: 12 & 10 is 8, because 12 is 1100 in binary, 10 is 1010, and only the 1000 bit is in both. Negative numbers are in two's complement, so -1 has every bit set: 12 & -1 is 12, and -1 & 255 is 255.

& binds more tightly than =, so perms & 2 = 2 is (perms & 2) = 2. On BIT values it works bit by bit as well; see Bitwise operators on BIT values.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT 12 & 10, 12 & -1, -1 & 255, 0 & 7, 5 & 3 = 1")
run("SELECT NULL & NULL, 1 & NULL, NULL & 1, 2147483647 & -1")
run("SELECT typeof(12::TINYINT & 10::TINYINT), typeof(12 & 10::BIGINT), "
    "(-1)::TINYINT & 255::UTINYINT, typeof((-1)::TINYINT & 255::UTINYINT)")
run(f"SELECT id, '12' & perms FROM {TABLE} ORDER BY id LIMIT 10")
run(f"SELECT 'abc' & perms FROM {TABLE} LIMIT 1")
run("SELECT 1.5 & 1")
run("SELECT true & false")

[(8, 12, 255, 0, True)]
[(None, None, None, 2147483647)]
[('TINYINT', 'BIGINT', 255, 'BIGINT')]
[(1, 4), (2, 4), (3, 0), (4, None)]
OSError: Could not convert string 'abc' to INT32
InternalError: Binder Error: No function matches the given name and argument types '&(DECIMAL(2,1), INTEGER_LITERAL)'. You might need to add explicit type casts.
InternalError: Binder Error: No function matches the given name and argument types '&(BOOLEAN, BOOLEAN)'. You might need to add explicit type casts.

Examples

The write (2) and admin (4) bits of each user. A bit that's set comes back as its value, one that isn't as 0:

run(f"SELECT id, name, perms & 2, perms & 4 FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 'ada', 2, 4), (2, 'grace', 0, 4), (3, 'alan', 0, 0), (4, None, None, None)]

Keep only the users who can write:

run(f"SELECT id, name FROM {TABLE} WHERE perms & 2 = 2 ORDER BY id LIMIT 10")

[(1, 'ada')]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not an operation on a column, so & on a column is always computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes & for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine computes & and keeps the rows that match. No rows are skipped in storage.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE perms & 2 = 2")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│     ((perms & 2) = 2)     │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│           perms           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Compare the whole mask when you can. A comparison of the raw column with a constant is pushed down to VAST DataBase, so rows that don't match never leave storage. A bitwise operation on constants is computed before the scan, so perms = (1 | 2 | 4) is pushed down too. The plan has no FILTER step:

plan(f"SELECT id FROM {TABLE} WHERE perms = (1 | 2 | 4)")

┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘

• Store the flags you filter on often. Keep a flag such as "can write" in its own BOOLEAN column when you write the rows: can_write = true is pushed down, perms & 2 = 2 isn't.

Limitations and errors

• Integers and BIT only. A DECIMAL, DOUBLE, FLOAT, BOOLEAN, DATE, INTERVAL, BLOB or list fails with No function matches the given name and argument types, for example '&(DOUBLE, INTEGER)'. Cast the value to an integer type first.
• No logical AND. true & false fails; use AND for booleans.
• No bitand function. bitand(12, 10) fails with Scalar Function with name bitand does not exist!. Use &.
• A string must be a whole number. A string next to an integer is converted to that integer's type at run time; 'abc' & perms fails with Could not convert string 'abc' to INT32.
• BIT results come back as bytes. The driver returns a BIT value as bytes; see Bitwise operators on BIT values.

run("SELECT 1.5::DOUBLE & 1")
run("SELECT CAST(1.5::DOUBLE AS INTEGER) & 1, true AND false")
run("SELECT bitand(12, 10)")

InternalError: Binder Error: No function matches the given name and argument types '&(DOUBLE, INTEGER_LITERAL)'. You might need to add explicit type casts.
[(0, False)]
InternalError: Catalog Error: Scalar Function with name bitand does not exist!

Similar functions

run("SELECT 12 & 10, 12 | 10, xor(12, 10), ~12, 12 << 1, 12 >> 1")

[(8, 14, 6, -13, 24, 6)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Operators and predicates: all the operators.
• Bitwise operators on BIT values.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.