json_exists

Applies to VAST Cluster 5.5.

Functions › Parsing and JSON functions › Check and inspect. Returns true if a path exists in a JSON document and false if it doesn't. A key whose value is JSON null exists.

Syntax

json_exists(json, path)
json_exists(json, paths)

Returns: BOOLEAN. A list of paths, or a path with a wildcard, returns BOOLEAN[].

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_json_json_exists. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four events with a JSON payload column: two objects, an array and a NULL. DOC is a sample document for the examples that don't read the table:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_json_json_exists"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.events"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, payload VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, '{\"user\": \"ada\", \"n\": 3, \"tags\": [\"a\", \"b\"], \"geo\": {\"city\": \"Oslo\", \"lat\": 59.9}}'), "
    "(2, '{\"user\": \"alan\", \"n\": \"many\", \"tags\": [], \"geo\": null}'), "
    "(3, '[10, 20, [30, 40]]'), "
    "(4, NULL)")

DOC = """'{"user": "ada", "n": 3, "tags": ["a", "b", "c"], "geo": {"city": "Oslo", "lat": 59.9}, "z": null}'"""

[]
[]
[(4,)]

What it does

json_exists follows the path with the same path forms as json_extract, and returns whether it found anything. It looks at the structure, not the value: a key that holds JSON null exists. With a wildcard or a list of paths, it returns one boolean per match or per path.

run(f"SELECT json_exists({DOC}, '$.user'), json_exists({DOC}, '$.geo.city'), json_exists({DOC}, '$.z'), "
    f"json_exists({DOC}, '$.missing')")
run(f"SELECT json_exists({DOC}, '/geo/city'), json_exists({DOC}, 'user'), json_exists({DOC}, '$.tags[-1]')")
run(f"SELECT json_exists({DOC}, '$.tags[*]'), json_exists({DOC}, ['$.user', '$.n', '$.missing'])")

[(True, True, True, False)]
[(True, True, True)]
[([True, True, True], [True, True, False])]

NULLs and edge cases

The same cases, as the cluster returns them:

run(f"SELECT json_exists(NULL, '$.user'), json_exists({DOC}, NULL), json_exists({DOC}, '$.z')")
run("SELECT json_exists('null', '$'), json_exists('[]', '$'), json_exists('[]', '$[0]'), "
    "json_exists('[1]', []::VARCHAR[])")
run("SELECT json_exists('not json', '$.a')")
run(f"SELECT json_exists({DOC}, 0)")

[(None, None, True)]
[(True, True, False, [])]
OSError: Malformed JSON at byte 0 of input: invalid literal.  Input: not json
InternalError: Binder Error: No function matches the given name and argument types 'json_exists(STRING_LITERAL, INTEGER_LITERAL)'. You might need to add explicit type casts.

Examples

Which events have a geo key, and which have a city inside it. Row 2's geo is JSON null: the key exists, the city doesn't:

run(f"SELECT id, json_exists(payload, '$.geo'), json_exists(payload, '$.geo.city') FROM {TABLE} ORDER BY id LIMIT 10")

[(1, True, True), (2, True, False), (3, False, False), (4, None, None)]

Keep the events that have a geo key:

run(f"SELECT id FROM {TABLE} WHERE json_exists(payload, '$.geo') ORDER BY id LIMIT 10")

[(1,), (2,)]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase stores a JSON document as text and can't look inside it, so json_exists is always computed by the Query Engine:

• In the SELECT list, the scan reads the whole payload column and the Query Engine parses each document it returns.
• In WHERE, the scan sends payload for every row, and a FILTER step in the Query Engine parses each document and keeps the rows that match. No rows are skipped in storage.
• Every document it reads must be valid JSON. The first invalid one stops the query with Malformed JSON ... (see invalid JSON in a column).

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE json_exists(payload, '$.geo')")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│  json_exists(payload, '$  │
│           .geo')          │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│          payload          │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Store a flag if you filter on it often. A BOOLEAN column such as has_geo, set when you write the row, is evaluated in VAST DataBase; json_exists(payload, '$.geo') parses every row in the Query Engine.
• Narrow the rows with a pushed-down condition. A condition such as id = 1 is evaluated in VAST DataBase, so a query that fails on the whole table because of one invalid document can still run on the rows it keeps.

Limitations and errors

• Paths are text. There's no integer form; write an array position as '$[0]'.
• No SQL-standard path modes. 'strict $.a' isn't read as a path mode: json_exists('{"a": 1}', 'strict $.a') returns false. Use a plain path.
• Invalid JSON stops the query. A pushed-down condition that leaves out the invalid rows lets the query run.
• A list of paths must be constant. A list from a column fails with List of paths must be constant; a list can't contain NULL (JSON path cannot be NULL) or a wildcard.
• The document must be text. A number fails with No function matches the given name and argument types; a BLOB fails with Malformed JSON ....

run("SELECT json_exists('{\"a\": 1}', 'strict $.a'), json_exists('{\"a\": 1}', '$.a')")
run(f"SELECT json_exists({DOC}, ['$.user', NULL])")

[(False, True)]
InternalError: Binder Error: JSON path cannot be NULL

Similar functions

Three ways to test for '$.z', a key whose value is JSON null:

run(f"SELECT json_exists({DOC}, '$.z'), json_extract({DOC}, '$.z') IS NOT NULL, "
    f"json_extract_string({DOC}, '$.z') IS NOT NULL")

[(True, True, False)]

To test whether a value occurs anywhere in a document, use json_contains. To list an object's keys, use json_keys.

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Parsing and JSON functions: all the JSON functions, and JSON paths.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.