regexp_extract_all

Applies to VAST Cluster 5.5.

Functions › Regular expressions › Regular expressions. regexp_extract_all(s, pattern) returns a list of every part of s that the regular expression pattern matches, from left to right. With a group number it returns that group of each match.

Syntax

regexp_extract_all(s, pattern)
regexp_extract_all(s, pattern, group)
regexp_extract_all(s, pattern, group, options)

Returns: VARCHAR[].

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_string_regexp_extract_all. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four request logs. Log 3 is empty, and the last row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_string_regexp_extract_all"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.requests"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, query VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, 'user=ada&id=17&id=42'), "
    "(2, 'Tag=SQL&tag=vast'), "
    "(3, ''), "
    "(4, NULL)")

[]
[]
[(4,)]

What it does

regexp_extract_all scans s from left to right. Each match starts where the previous one ended, so matches don't overlap: regexp_extract_all('aaa', 'aa') is ['aa'].

Without a match, the result is an empty list []. A pattern that can match an empty string adds an empty string at each place it matches nothing: regexp_extract_all('baaac', 'a*') is ['', 'aaa', '', '']. To count the matches, take the length of the list.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT regexp_extract_all(NULL, 'a'), regexp_extract_all('abc', NULL), regexp_extract_all('abc', '(a)', NULL)")
run("SELECT regexp_extract_all('abc', 'a', 0, NULL)")
run("SELECT regexp_extract_all('abc', 'x'), regexp_extract_all('', 'a'), regexp_extract_all('abc', ''), "
    "regexp_extract_all('', '')")
run("SELECT regexp_extract_all('ab', '(x)?(b)', 1), regexp_extract_all('a1b22', '([a-z])(\\d+)', -1), "
    "len(regexp_extract_all('e\u0301e\u0301', '.'))")
run("SELECT regexp_extract_all('a1b22', '([a-z])(\\d+)', 3)")
run("SELECT regexp_extract_all('abc', '(')")

[(None, None, None)]
InternalError: Invalid Input Error: Regex options field must not be NULL
[([], [], ['', '', '', ''], [''])]
[([None], [], 4)]
OSError: Pattern has 2 groups. Cannot access group 3
OSError: Pattern failed to parse, error: 'missing ): ('

Examples

Every number in each query string, every id value (group 2 of (id)=(\w+)), and the number of parameters:

run(f"SELECT id, regexp_extract_all(query, '\\d+'), regexp_extract_all(query, '(id)=(\\w+)', 2), "
    f"len(regexp_extract_all(query, '\\w+=')) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, ['17', '42'], ['17', '42'], 3), (2, [], [], 2), (3, [], [], 0), (4, None, None, None)]

The tag values, in any case, with option 'i':

run(f"SELECT id, regexp_extract_all(query, 'tag=(\\w+)', 1), regexp_extract_all(query, 'tag=(\\w+)', 1, 'i') "
    f"FROM {TABLE} ORDER BY id LIMIT 10")

[(1, [], []), (2, ['vast'], ['SQL', 'vast']), (3, [], []), (4, None, None)]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). regexp_extract_all is always evaluated in the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine builds a list for each value it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step computes regexp_extract_all() and keeps the rows that match.

plan(f"SELECT id FROM {TABLE} WHERE len(regexp_extract_all(query, '\\w+=')) > 2")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│  (len(regexp_extract_all  │
│   (query, '\w+=')) > 2)   │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│           query           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Test for a match with regexp_matches, not with len(regexp_extract_all(...)) > 0: it returns a BOOLEAN without building a list, and simple patterns are rewritten into cheaper tests.
• Extract at load time. If you filter on the extracted values often, store them in their own column, such as a VARCHAR[], when you write the data.
• Combine it with a pushed-down condition. In id > 100 AND ..., id > 100 is still evaluated in VAST DataBase, so the list is built only for the rows that pass it.

Limitations and errors

• The group must exist in the pattern. A larger number fails with Pattern has N groups. Cannot access group M. A negative group gives [].
• The options string is a constant and isn't NULL; option g fails with Option 'g' (global replace) is only valid for regexp_replace.
• No regexp_count. It fails with Scalar Function with name regexp_count does not exist!. Use len(regexp_extract_all(s, pattern)).

run("SELECT regexp_extract_all('ABab', 'a', 0, 'g')")
run("SELECT regexp_count('abab', 'a')")
run("SELECT len(regexp_extract_all('abab', 'a'))")

InternalError: Invalid Input Error: Option 'g' (global replace) is only valid for regexp_replace
InternalError: Catalog Error: Scalar Function with name regexp_count does not exist!
[(2,)]

Similar functions

run("SELECT regexp_extract_all('a,b,,c', '[^,]+'), regexp_split_to_array('a,b,,c', ','), "
    "string_split('a,b,,c', ','), regexp_extract('a,b,,c', '[^,]+')")

[(['a', 'b', 'c'], ['a', 'b', '', 'c'], ['a', 'b', '', 'c'], 'a')]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Regular-expression syntax and options.
• Regular expressions: all the regular-expression functions.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.