array_length

Applies to VAST Cluster 5.5.

Functions › List, struct and map functions › Size and elements. Returns the number of elements in a list, NULL elements included. For a list of lists, it counts the outer list. length and len return the same count.

Syntax

array_length(list)
array_length(list, dimension)

Returns: BIGINT.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_lists_array_length. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four survey responses: a list of answers, one of them unanswered (NULL) in the second row, and a grid of answers as a list of lists. The third row has empty lists and the last row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_lists_array_length"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.surveys"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, answers BIGINT[], grid BIGINT[][])")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, [4, 5, 3], [[1, 2], [3]]), "
    "(2, [2, NULL, 5], [[], NULL, [4]]), "
    "(3, [], []), "
    "(4, NULL, NULL)")

[]
[]
[(4,)]

What it does

array_length(l) counts the top-level elements of l:

• NULL elements count. array_length([1, NULL, 3]) is 3. To count only the non-NULL elements, use list_count.
• Only the outer list. array_length([[1, 2], [3]]) is 2, the number of inner lists, whatever their lengths.
• The dimension must be 1. array_length(l, 1) is the same as array_length(l). Any other dimension fails; see Limitations and errors.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT array_length(NULL), array_length([1, 2], NULL), array_length([]), array_length([1, NULL, 3]), "
    "array_length([[1, 2], [3]]), array_length(array_value(1, 2, 3))")
run("SELECT array_length([[1, 2], [3]], 2)")
run("SELECT array_length('abc')")

[(None, None, 0, 3, 2, 3)]
OSError: array_length for lists with dimensions other than 1 not implemented
InternalError: Binder Error: No function matches the given name and argument types 'array_length(STRING_LITERAL)'. You might need to add explicit type casts.

Examples

The number of answers and of grid rows in each response. The NULL answer in response 2 is counted:

run(f"SELECT id, array_length(answers), array_length(answers, 1), array_length(grid) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 3, 3, 2), (2, 3, 3, 3), (3, 0, 0, 0), (4, None, None, None)]

Keep the responses with more than one answer:

run(f"SELECT id FROM {TABLE} WHERE array_length(answers) > 1 ORDER BY id LIMIT 10")

[(1,), (2,)]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). A list column is read whole, and every condition on a list runs in the Query Engine, so array_length() is always computed there:

• In the SELECT list, the scan reads the list and the Query Engine counts it for each row it returns.
• In WHERE, the scan sends the list for every row, and a FILTER step in the Query Engine counts it and keeps the rows that match. No rows are skipped in storage.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE array_length(answers) > 1")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│(array_length(answers) > 1)│
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│          answers          │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• An empty-list test isn't faster. answers = [] and answers IS NULL also run in a FILTER step: no condition on a list column is pushed down.
• Store the count if you filter on it often. Compute it when you write the rows and keep it in its own column: answer_count > 1 is pushed down to VAST DataBase, array_length(answers) > 1 isn't.
• Combine it with a pushed-down condition. In id > 100 AND array_length(answers) > 1, id > 100 is still evaluated in VAST DataBase, so array_length() runs only on the rows that pass it.

plan(f"SELECT id FROM {TABLE} WHERE answers = []")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│       (answers = [])      │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│          answers          │
│             id            │
│                           │
└───────────────────────────┘

Limitations and errors

• Only the first dimension. array_length(l, 2) fails with array_length for lists with dimensions other than 1 not implemented, as do 0 and negative dimensions. To get the length of the longest inner list, use list_transform and list_max: list_max(list_transform(grid, x -> len(x))). It returns NULL for an empty list.
• Lists only. A string, a number, a BLOB, a date or a boolean fails with No function matches the given name and argument types. For the length of a string, use length.
• Other SQL dialects' names aren't available. cardinality works only on maps (Binder Error: Cardinality can only operate on MAPs), and array_size fails with Scalar Function with name array_size does not exist!. Use array_length or len.

run(f"SELECT id, list_max(list_transform(grid, x -> len(x))) FROM {TABLE} ORDER BY id LIMIT 10")
run("SELECT cardinality([1, 2, 3])")
run("SELECT array_size([1, 2])")

[(1, 2), (2, 1), (3, None), (4, None)]
InternalError: Binder Error: Cardinality can only operate on MAPs
InternalError: Catalog Error: Scalar Function with name array_size does not exist!

Similar functions

run("SELECT array_length(l), array_length(l, 1), len(l), length(l), list_count(l), list_unique(l) "
    "FROM (SELECT [3, 1, NULL, 3, 10] AS l)")

[(5, 5, 5, 5, 4, 3)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• List, struct and map functions: all the list functions.
• length, list_count and list_unique.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.