list_extract

Applies to VAST Cluster 5.5.

Functions › List, struct and map functions › Size and elements. Returns the element of a list at a position, counting from 1; a negative position counts from the end. list_element and array_extract are the same function, and l[i] is the bracket form of it. On a VARCHAR, it returns one character.

Syntax

list_extract(list, index)
list_element(list, index)
array_extract(list, index)
list[index]

Returns: the list's element type; VARCHAR for a string.

Aliases: list_element, array_extract, list[index].

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_lists_list_extract. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four orders: the items, their quantities, a position to pick and a code. The second order has a NULL item, the third has empty lists and the last row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_lists_list_extract"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.orders"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, items VARCHAR[], qty BIGINT[], pick BIGINT, code VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, ['pen', 'ink', 'pad'], [2, 1, 5], 2, 'AB7'), "
    "(2, ['cup', NULL], [1, NULL], -1, 'Q'), "
    "(3, [], [], 1, ''), "
    "(4, NULL, NULL, NULL, NULL)")

[]
[]
[(4,)]

What it does

list_extract(l, i) returns the element at position i of l. Positions count from 1, and a negative position counts from the end:

A position that doesn't exist returns NULL, not an error. So does a position that holds a NULL element: list_extract([1, NULL, 3], 2) is NULL as well. The index can be a column: items[pick] takes a different position in each row.

On a string, list_extract(s, i) returns the character at position i as a VARCHAR: list_extract('abc', 2) is 'b' and list_extract('abc', -1) is 'c'. A position that doesn't exist returns an empty string '', not NULL. For more than one character, use substring.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT list_extract(NULL, 2), list_extract([1, 2], NULL), list_extract([], 1), list_extract([1, NULL, 3], 2)")
run("SELECT list_extract([1, 2, 3], 0), list_extract([1, 2, 3], 4), list_extract([1, 2, 3], -4), list_extract([1, 2, 3], '2')")
run("SELECT list_extract('abc', 2), list_extract('abc', -1), list_extract('abc', 0), list_extract('abc', 9)")
run("SELECT list_extract([1, 2, 3], 1.5)")
run("SELECT list_extract(123, 2)")

[(None, None, None, None)]
[(None, None, None, 2)]
[('b', 'c', '', '')]
InternalError: Binder Error: No function matches the given name and argument types 'list_extract(INTEGER[], DECIMAL(2,1))'. You might need to add explicit type casts.
InternalError: Binder Error: No function matches the given name and argument types 'list_extract(INTEGER_LITERAL, INTEGER_LITERAL)'. You might need to add explicit type casts.

Examples

The first and the last item of each order, with each of the four forms:

run(f"SELECT id, items[1], list_extract(items, -1), list_element(items, 1), array_extract(items, -1) "
    f"FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 'pen', 'pad', 'pen', 'pad'), (2, 'cup', None, 'cup', None), (3, None, None, None, None), (4, None, None, None, None)]

The item and the quantity at the position in pick, and the first character of the code. Order 2 picks position -1, its last item, which is NULL:

run(f"SELECT id, items[pick], qty[pick], list_extract(code, 1) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 'ink', 1, 'A'), (2, None, None, 'Q'), (3, None, None, ''), (4, None, None, None)]

Keep the orders whose first item is a pen:

run(f"SELECT id FROM {TABLE} WHERE items[1] = 'pen' ORDER BY id LIMIT 10")

[(1,)]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). A list column is read whole, and every condition on a list runs in the Query Engine, so list_extract() and l[i] are always computed there:

• In the SELECT list, the scan reads the list and the Query Engine takes the element for each row it returns.
• In WHERE, the scan sends the whole list for every row, and a FILTER step in the Query Engine takes the element and keeps the rows that match. No rows are skipped in storage.

The plan of the WHERE example shows the FILTER step above the scan. The plan writes items[1] as array_extract(items, 1):

plan(f"SELECT id FROM {TABLE} WHERE items[1] = 'pen'")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│ (array_extract(items, 1) =│
│           'pen')          │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│           items           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Store an element you filter on in its own column. If queries often test the first item, keep it in a column such as first_item when you write the rows: first_item = 'pen' is pushed down to VAST DataBase, so rows that don't match never leave storage, while items[1] = 'pen' runs in a FILTER step. The plan of a plain column compared with a constant, code = 'AB7', has no FILTER step:

plan(f"SELECT id FROM {TABLE} WHERE code = 'AB7'")

┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘

• Combine it with a pushed-down condition. In id > 100 AND items[1] = 'pen', id > 100 is still evaluated in VAST DataBase, so the element is taken only from the rows that pass it.

Limitations and errors

• Positions must be integers. A DECIMAL such as 1.5 fails with No function matches the given name and argument types; a string that isn't a number fails with Could not convert string 'a' to INT64. Cast it to an integer first: list_extract(l, CAST(x AS BIGINT)).
• NULL means missing or NULL. A position that doesn't exist and a NULL element both return NULL. Compare the position with array_length if you need to tell them apart.
• Not for maps or structs. Use map_extract or m[key] for a map, and struct_extract or s.field for a struct. element_at works only on maps: on a list it fails with Binder Error: 'element_at' can only operate on MAPs.

run("SELECT list_extract([1, 2, 3], 'a')")
run("SELECT element_at([1, 2, 3], 1)")

OSError: Could not convert string 'a' to INT64
InternalError: Binder Error: 'element_at' can only operate on MAPs

Similar functions

run("SELECT l[2], list_extract(l, 2), list_element(l, 2), array_extract(l, 2), l[-1], l[2:2], list_slice(l, 2, 2), "
    "list_select(l, [2]), list_first(l), list_last(l) FROM (SELECT [3, 1, NULL, 3, 10] AS l)")

[(1, 1, 1, 1, 10, [1], [1], [1], 3, 10)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• List, struct and map functions: all the list functions.
• list_slice, array_length and substring.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.