json_keys

Applies to VAST Cluster 5.5.

Functions › Parsing and JSON functions › Check and inspect. Returns the keys of a JSON object as a list, in the order they appear in the document. For an array, a scalar or JSON null, it returns an empty list.

Syntax

json_keys(json)
json_keys(json, path)
json_keys(json, paths)

Argument	Type	Description
`json`	`VARCHAR` or `JSON`	The JSON document.
`path`	`VARCHAR`	A JSON path, JSON pointer or top-level key, to list the keys of an inner object.
`paths`	`VARCHAR[]`	A constant list of paths.

Returns: VARCHAR[]. A list of paths returns VARCHAR[][], one list of keys per path.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_json_json_keys. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four events with a JSON payload column: two objects, an array and a NULL. DOC is a sample document for the examples that don't read the table:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_json_json_keys"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.events"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, payload VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, '{\"user\": \"ada\", \"n\": 3, \"tags\": [\"a\", \"b\"], \"geo\": {\"city\": \"Oslo\", \"lat\": 59.9}}'), "
    "(2, '{\"user\": \"alan\", \"n\": \"many\", \"tags\": [], \"geo\": null}'), "
    "(3, '[10, 20, [30, 40]]'), "
    "(4, NULL)")

DOC = """'{"user": "ada", "n": 3, "tags": ["a", "b", "c"], "geo": {"city": "Oslo", "lat": 59.9}, "z": null}'"""

[]
[]
[(4,)]

What it does

json_keys parses the document, follows the path if you give one, and returns the top-level keys of the object it finds. It doesn't go into nested objects; give a path for those. Keys come in document order, not sorted, and a key that appears twice is listed twice.

run(f"SELECT json_keys({DOC}), json_keys({DOC}, '$.geo')")
run("SELECT json_keys('{\"b\": 1, \"a\": 2}'), json_keys('{\"a\": 1, \"a\": 2}')")

[(['user', 'n', 'tags', 'geo', 'z'], ['city', 'lat'])]
[(['b', 'a'], ['a', 'a'])]

NULLs and edge cases

Input	Result
A NULL document or a NULL path	`NULL`
A path that isn't in the document (`'$.missing'`)	`NULL`
An array, a string, a number, `true`, JSON `null` or `{}`	`[]`
An empty key, `{"": 2}`	`''` is listed
A list of paths	one list per path, `NULL` for a missing one
`'not json'`	Error: `Malformed JSON at byte 0 of input: invalid literal. Input: not json`
An integer path, `json_keys(json, 0)`	Error: `No function matches the given name and argument types`; write `'$[0]'`

The same cases, as the cluster returns them:

run(f"SELECT json_keys(NULL), json_keys({DOC}, NULL), json_keys({DOC}, '$.missing')")
run(f"SELECT json_keys({DOC}, '$.tags'), json_keys({DOC}, '$.user'), json_keys({DOC}, '$.n'), json_keys({DOC}, '$.z'), "
    "json_keys('{}')")
run("SELECT json_keys('{\"x\": 1, \"\": 2}')")
run(f"SELECT json_keys({DOC}, ['$.geo', '$.tags', '$.missing'])")
run("SELECT json_keys('not json')")
run(f"SELECT json_keys({DOC}, 0)")

[(None, None, None)]
[([], [], [], [], [])]
[(['x', ''],)]
[([['city', 'lat'], [], None],)]
OSError: Malformed JSON at byte 0 of input: invalid literal.  Input: not json
InternalError: Binder Error: No function matches the given name and argument types 'json_keys(STRING_LITERAL, INTEGER_LITERAL)'. You might need to add explicit type casts.

Examples

The keys of each document and of its geo object. Row 2's geo is JSON null, so its keys are an empty list; the array in row 3 has no geo:

run(f"SELECT id, json_keys(payload), json_keys(payload, '$.geo') FROM {TABLE} ORDER BY id LIMIT 10")

[(1, ['user', 'n', 'tags', 'geo'], ['city', 'lat']), (2, ['user', 'n', 'tags', 'geo'], []), (3, [], None), (4, None, None)]

Keep the documents that have a geo key, with list_contains:

run(f"SELECT id FROM {TABLE} WHERE list_contains(json_keys(payload), 'geo') ORDER BY id LIMIT 10")

[(1,), (2,)]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase stores a JSON document as text and can't look inside it, so json_keys is always computed by the Query Engine:

In the SELECT list, the scan reads the whole payload column and the Query Engine parses each document it returns.
In WHERE, the scan sends payload for every row, and a FILTER step in the Query Engine parses each document and keeps the rows that match. No rows are skipped in storage.
Every document it reads must be valid JSON. The first invalid one stops the query with Malformed JSON ... (see invalid JSON in a column).

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE list_contains(json_keys(payload), 'geo')")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│  list_contains(json_keys  │
│     (payload), 'geo')     │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│          payload          │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

Test one key with json_exists. json_exists(payload, '$.geo') answers the same question as list_contains(json_keys(payload), 'geo') directly. Both run in the Query Engine.
Store the fields you filter on in their own columns. A comparison of a column with a constant is evaluated in VAST DataBase.
Narrow the rows with a pushed-down condition. A condition such as id = 1 is evaluated in VAST DataBase, so a query that fails on the whole table because of one invalid document can still run on the rows it keeps.

Limitations and errors

Top-level keys only. Give a path to list the keys of an inner object.
Duplicate keys are kept. json_keys('{"a": 1, "a": 2}') returns ['a', 'a'].
Paths are text. There's no integer form; write an array position as '$[0]'.
Invalid JSON stops the query. Use json_valid to find invalid documents first.
A list of paths must be constant. A list from a column fails with List of paths must be constant.
The document must be text. A number fails with No function matches the given name and argument types, for example 'json_keys(INTEGER_LITERAL)'; a BLOB fails with Malformed JSON ....

run("SELECT json_keys(123)")

InternalError: Binder Error: No function matches the given name and argument types 'json_keys(INTEGER_LITERAL)'. You might need to add explicit type casts.

Similar functions

Function	`'{"geo": {"city": "Oslo"}, "z": null}'`
`json_keys`	`['geo', 'z']`: top-level keys
json_exists with `'$.z'`	`true`: one key, also when its value is `null`
json_structure	`'{"geo":{"city":"VARCHAR"},"z":"NULL"}'`: every key, at every depth, with its type
json_type	`'OBJECT'`

run("SELECT json_keys('{\"geo\": {\"city\": \"Oslo\"}, \"z\": null}'), "
    "json_exists('{\"geo\": {\"city\": \"Oslo\"}, \"z\": null}', '$.z'), "
    "json_structure('{\"geo\": {\"city\": \"Oslo\"}, \"z\": null}'), "
    "json_type('{\"geo\": {\"city\": \"Oslo\"}, \"z\": null}')")

[(['geo', 'z'], True, '{"geo":{"city":"VARCHAR"},"z":"NULL"}', 'OBJECT')]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]