split_part

Applies to VAST Cluster 5.5.

Functions › Search and split › Search and split. Splits a string on a separator and returns one field: split_part('a,b,c', ',', 2) is 'b'. Fields count from 1; a negative position counts from the end. A field that doesn't exist gives an empty string, not NULL.

Syntax

split_part(s, separator, n)

Returns: VARCHAR.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_string_split_part. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four log files, each with a path, an owner's email address and a comma-separated list of tags. The last row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_string_split_part"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.logs"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, path VARCHAR, owner VARCHAR, tags VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, '/var/log/app/server.log', 'ada@example.com', 'prod,web'), "
    "(2, '/var/log/db/query.log', 'grace@example.org', 'prod,db,eu'), "
    "(3, '/tmp/debug.log', 'jos\u00e9@example.co.uk', 'dev,,local'), "
    "(4, NULL, NULL, NULL)")

[]
[]
[(4,)]

What it does

split_part(s, separator, n) cuts s at every occurrence of separator and returns field n:

A path that starts with the separator has an empty first field: in '/var/log/app' split on '/', field 1 is '' and field 2 is 'var'.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT split_part(NULL, ',', 2), split_part('a,b', NULL, 1), split_part('a,b', ',', NULL)")
run("SELECT split_part('a,b,c', ',', 0), split_part('a,b,c', ',', 4), split_part('a,b,c', ',', -4), split_part('a,b', ',', 9223372036854775807)")
run("SELECT split_part('a,,c', ',', 2), split_part('a,b', 'x', 1), split_part('a,b', 'x', 2), split_part('', ',', 1)")
run("SELECT split_part('A,b', 'a', 1), split_part('\u00e9,\u00fc', ',', 2)")
run("SELECT split_part('a,b', ',', 1.7)")

[(None, None, None)]
[('', '', '', '')]
[('', 'a,b', '', '')]
[('A,b', 'ü')]
InternalError: Binder Error: No function matches the given name and argument types 'array_extract(VARCHAR[], DECIMAL(2,1))'. You might need to add explicit type casts.

Examples

The directory under /var/log, the owner's domain (the last field after @) and the third tag of each file:

run(f"SELECT id, split_part(path, '/', 4), split_part(owner, '@', -1), split_part(tags, ',', 3) "
    f"FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 'app', 'example.com', ''), (2, 'db', 'example.org', 'eu'), (3, '', 'example.co.uk', 'local'), (4, None, None, None)]

The files whose first tag is prod:

run(f"SELECT id, path FROM {TABLE} WHERE split_part(tags, ',', 1) = 'prod' ORDER BY id LIMIT 10")

[(1, '/var/log/app/server.log'), (2, '/var/log/db/query.log')]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so split_part() is always computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes split_part() for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine computes split_part() and keeps the rows that match. No rows are skipped in storage.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE split_part(tags, ',', 1) = 'prod'")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│ (CASE  WHEN ((tags IS NOT │
│    NULL)) THEN (COALESCE  │
│       (array_extract      │
│ (string_split(tags, ','), │
│  1), '')) ELSE NULL END = │
│          'prod')          │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│            tags           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Store the fields you filter on. If you often split a column to filter on one field, write that field to its own column when you load the data. A comparison of the raw column with a constant is pushed down to VAST DataBase, so rows that don't match never leave storage. The plan for owner = 'ada@example.com' has no FILTER step:

plan(f"SELECT id FROM {TABLE} WHERE owner = 'ada@example.com'")

┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘

• Combine it with a pushed-down condition. In id > 100 AND split_part(tags, ',', 1) = 'prod', id > 100 is still evaluated in VAST DataBase, so split_part() runs only on the rows that pass it.

Limitations and errors

• Strings only. A number, date, list or BLOB fails with an error that names string_split, which split_part calls: for example No function matches the given name and argument types 'string_split(INTEGER_LITERAL, STRING_LITERAL)'. Cast the value to VARCHAR first.
• n must be an integer. A decimal fails with the array_extract error shown above.
• An empty string, not NULL, for a missing field. Test with = '', or use NULLIF(split_part(s, sep, n), '') to turn it into NULL.
• The separator is plain text. For a regular expression, use regexp_split_to_array.

run("SELECT split_part(123, ',', 2)")
run("SELECT split_part(CAST(123 AS VARCHAR), '2', 2), NULLIF(split_part('a,b', ',', 3), '')")

InternalError: Binder Error: No function matches the given name and argument types 'string_split(INTEGER_LITERAL, STRING_LITERAL)'. You might need to add explicit type casts.
[('3', None)]

Similar functions

run("SELECT split_part('a,b,c', ',', 2), string_split('a,b,c', ',')[2], str_split('a,b,c', ',')[2]")
run("SELECT split_part('a,b,c', ',', 5), string_split('a,b,c', ',')[5]")

[('b', 'b', 'b')]
[('', None)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Search and split: all the search and split functions.
• string_split and regexp_split_to_array.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.