url_decode

Applies to VAST Cluster 5.5.

Functions › Text basics › Unicode and URLs. Decodes a percent-encoded string: each % followed by two hexadecimal digits becomes the byte it stands for, and the bytes are read as UTF-8. It reverses url_encode. It has no aliases.

Syntax

url_decode(s)

Returns: VARCHAR.

Aliases: none.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_url_decode. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of query-string values from a web server's access log, still percent-encoded. Row 3 came from a form, so its space is a +. The last row has no value:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_url_decode"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.requests"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, param VARCHAR, value VARCHAR)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, 'q', 'caf%C3%A9%20menu'), "
    "(2, 'path', '%2Fdata%2Flogs%2Fapp.log'), "
    "(3, 'name', 'Ada+Lovelace'), "
    "(4, 'q', 'R%26D'), "
    "(5, 'q', NULL)")

[]
[]
[(5,)]

What it does

url_decode reads the string from left to right. Each % followed by two hexadecimal digits is replaced by the byte with that value; everything else is copied as it is. The resulting bytes must be valid UTF-8:

Text that isn't a valid escape is left as it is, rather than causing an error:

+ stays +. HTML forms and some query strings encode a space as + (application/x-www-form-urlencoded). url_decode doesn't, so for those values replace + with a space first: url_decode(replace(s, '+', ' ')) (Examples shows it). Do the replacement before decoding, so an encoded plus (%2B) still decodes to +.

Invalid UTF-8 is an error. url_decode('%E9') decodes to the single byte E9, which isn't valid UTF-8 on its own (it's é in Latin-1), so the query fails with Failed to decode string "%E9" using URL decoding - decoded value is invalid UTF8. On a column, one such row stops the whole query, and it returns no rows, including the rows that would have decoded.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT url_decode(NULL), url_decode(''), url_decode('a+b'), url_decode('%zz'), url_decode('%'), "
    "url_decode('%C3%A9')")
run("SELECT url_decode('%E9')")
run("SELECT url_decode(123)")

[(None, '', 'a+b', '%zz', '%', 'é')]
OSError: Failed to decode string "%E9" using URL decoding - decoded value is invalid UTF8
InternalError: Binder Error: No function matches the given name and argument types 'url_decode(INTEGER_LITERAL)'. You might need to add explicit type casts.

Examples

Each value decoded. Row 3 keeps its +:

run(f"SELECT id, param, url_decode(value) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 'q', 'café menu'), (2, 'path', '/data/logs/app.log'), (3, 'name', 'Ada+Lovelace'), (4, 'q', 'R&D'), (5, 'q', None)]

For form-encoded values, replace + with a space before decoding:

run(f"SELECT id, url_decode(replace(value, '+', ' ')) FROM {TABLE} WHERE param = 'name' ORDER BY id LIMIT 10")

[(3, 'Ada Lovelace')]

Find the requests for files under /data/, by their decoded path:

run(f"SELECT id, url_decode(value) FROM {TABLE} WHERE url_decode(value) LIKE '/data/%' ORDER BY id LIMIT 10")

[(2, '/data/logs/app.log')]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so url_decode() is always computed by the Query Engine:

• In the SELECT list, the scan reads the column and the Query Engine computes url_decode() for each row it returns.
• In WHERE, the scan sends the column for every row, and a FILTER step in the Query Engine computes url_decode() and keeps the rows that match. No rows are skipped in storage.

The plan of the last example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE url_decode(value) LIKE '/data/%'")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│ prefix(url_decode(value), │
│         '/data/')         │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│           value           │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Decode once, when you write. If you query the decoded values often, decode them in the application that loads the rows and store the decoded text. Then a comparison of the raw column with a constant is pushed down to VAST DataBase, so rows that don't match never leave storage.
• Decode the constant, not the column. Once the column holds decoded text, decode the encoded value you search for, not the column. In query_text = url_decode('caf%C3%A9%20menu'), url_decode() of a constant is computed once, and the comparison of the raw column with the result is pushed down.
• Narrow the rows with a pushed-down condition first. In param = 'path' AND url_decode(value) LIKE '/data/%', param = 'path' is evaluated in VAST DataBase, so url_decode() runs only on the rows that pass it. The plan for param = 'path' alone has no FILTER step:

plan(f"SELECT id FROM {TABLE} WHERE param = 'path'")

┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘

Limitations and errors

• Escapes that don't decode to valid UTF-8 fail. %E9 alone fails with Failed to decode string "%E9" using URL decoding - decoded value is invalid UTF8, and on a column one such row stops the whole query. Text encoded from Latin-1 or another single-byte character set can't be decoded with url_decode.
• + isn't a space. See What it does.
• Strings only. A number, decimal, date, boolean, BLOB or list fails with No function matches the given name and argument types.

run("SELECT url_decode('caf%E9')")
run("SELECT url_decode('\\xAA'::BLOB)")

OSError: Failed to decode string "caf%E9" using URL decoding - decoded value is invalid UTF8
InternalError: Binder Error: No function matches the given name and argument types 'url_decode(BLOB)'. You might need to add explicit type casts.

Similar functions

run("SELECT url_decode('a%20b'), url_encode('a b'), url_decode(replace('a+b', '+', ' '))")

[('a b', 'a%20b', 'a b')]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Text basics: all the text functions.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.
• url_encode and replace.