from_binary

Applies to VAST Cluster 5.5.

Functions › Binary, hash and UUID functions › Binary text. Reads a string of 0s and 1s and returns the bytes it describes, as a BLOB. It's the reverse of bin. unbin is the same function.

Syntax

from_binary(s)
unbin(s)

Returns: BLOB.

Aliases: unbin.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_string_from_binary. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of bit strings. The last row has no value:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_string_from_binary"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.bits"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, b VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, '01100001'), "
    "(2, '0110100001101001'), "
    "(3, '0110'), "
    "(4, '101010101'), "
    "(5, ''), "
    "(6, NULL)")

[]
[]
[(6,)]

What it does

from_binary reads the digits eight at a time and returns one byte for each group. If the number of digits isn't a multiple of eight, it's padded on the left with zeros to whole bytes, so the digits keep their value:

The result is a BLOB, which the Python driver returns as bytes. To turn bytes that hold UTF-8 text back into a VARCHAR, use decode: decode(from_binary(bin('café'))) is 'café'.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT from_binary(NULL), from_binary(''), from_binary('0'), from_binary('1'), "
    "octet_length(from_binary('111111111')), unbin('01100001')")
run("SELECT from_binary('012')")
run("SELECT from_binary('0110 0001')")
run("SELECT from_binary('0b0110')")
run("SELECT from_binary(1010)")

[(None, b'', b'\x00', b'\x01', 2, b'a')]
OSError: Invalid input for binary digit: 2
OSError: Invalid input for binary digit:  
OSError: Invalid input for binary digit: b
InternalError: Binder Error: No function matches the given name and argument types 'from_binary(INTEGER_LITERAL)'. You might need to add explicit type casts.

Examples

Each bit string as bytes, and the number of bytes:

run(f"SELECT id, b, from_binary(b), unbin(b), octet_length(from_binary(b)) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, '01100001', b'a', b'a', 1), (2, '0110100001101001', b'hi', b'hi', 2), (3, '0110', b'\x06', b'\x06', 1), (4, '101010101', b'\x01U', b'\x01U', 2), (5, '', b'', b'', 0), (6, None, None, None, None)]

A round trip through bin: the bytes of a string, and the string again:

run("SELECT from_binary(bin('hi')), decode(from_binary(bin('caf\u00e9'))), from_binary(bin(5))")

[(b'hi', 'café', b'\x05')]

Keep the rows that make exactly one byte:

run(f"SELECT id, b FROM {TABLE} WHERE octet_length(from_binary(b)) = 1 ORDER BY id LIMIT 10")

[(1, '01100001'), (3, '0110')]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). from_binary() is a function of a column, so it's computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes from_binary() for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine computes from_binary() and keeps the rows that match.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE octet_length(from_binary(b)) = 1")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│ (octet_length(from_binary │
│         (b)) = 1)         │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│             b             │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Store bytes as a BLOB, not as digits. A BLOB column takes an eighth of the space of its digits, and needs no conversion when you read it.
• Combine it with a pushed-down condition. In id > 100 AND octet_length(from_binary(b)) = 1, id > 100 can be evaluated in VAST DataBase, so from_binary() runs only on the rows that pass it. See pushdowns.

Limitations and errors

• Only 0 and 1. Any other character, including a space or a 0b prefix, stops the query with an OSError: Invalid input for binary digit: <character>. Clean the digits first, for example with replace to remove spaces.
• VARCHAR only. A number fails with No function matches the given name and argument types; write the digits as a string ('1010'), or use bin to make them.

Similar functions

run("SELECT from_binary('01100001'), unbin('01100001'), from_hex('61'), bin('a'), to_binary('a')")

[(b'a', b'a', b'a', '01100001', '01100001')]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Binary, hash and UUID functions: all the binary and encoding functions.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.