log

Applies to VAST Cluster 5.5.

Functions › Numeric and math functions › Powers and logarithms. With one argument, returns the base-10 logarithm of a number; with two, log(b, x) returns the logarithm of x to base b. The result is a DOUBLE. Zero and negative numbers fail the query.

Syntax

log(x)
log(b, x)

Returns: DOUBLE, whatever the argument types.

Set up the examples

The examples connect with the ADBC driver, as in VAST Query Engine (ADBC), and work in a schema called docs_sql_fn_numeric_log. Set QE_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and VASTDB_BUCKET (SQL reference), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the run() and plan() helpers, and creates a table of four values x, each with a base b. The last row has no values:

import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_numeric_log"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.measures"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, b DOUBLE, x DOUBLE)")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, 2.0, 8.0), "
    "(2, 10.0, 1000.0), "
    "(3, 0.5, 8.0), "
    "(4, 3.0, 1.0), "
    "(5, NULL, NULL)")

[]
[]
[(5,)]

What it does

log(x) is the base-10 logarithm, the same as log10: log(100) is 2.0 and log(1e308) is 308.0. It isn't the natural logarithm, as it is in some other SQL dialects: log(exp(1)) is 0.4342944819032518. For base e, use ln.

log(b, x) takes the base first. log(2, 8) is 3.0, because 2³ is 8; log(8, 2) is 0.33333333333333337. A base between 0 and 1 gives the opposite sign: log(0.5, 8) is -3.0.

The arguments are converted to DOUBLE first, so integers, DECIMAL and FLOAT values all work and all give a DOUBLE; so does a string that holds a number.

Values a logarithm can't take are errors, not infinity or NaN, and they stop the whole query:

• 0 as x or as the base fails with cannot take logarithm of zero. A negative zero (-0.0::DOUBLE) counts as zero.
• A negative number as x or as the base fails with cannot take logarithm of a negative number; so does '-inf'::DOUBLE.
• A base of 1 fails with divison by zero in based logarithm (the server's spelling).

log('inf'::DOUBLE) is inf, and nan as either argument gives nan. An infinite base gives 0: log('inf'::DOUBLE, 2) is 0.0.

NULLs and edge cases

The same cases, as the cluster returns them. They use the table and helpers from Set up the examples:

run("SELECT log(NULL), log(2, NULL), log(NULL, 8), log(1), log(100), log(1e308), log(5e-324::DOUBLE)")
run("SELECT log('inf'::DOUBLE), log('nan'::DOUBLE), log(2, 8), log(8, 2), log(10, 1000), log(0.5, 8), log(2, 1)")
run("SELECT log(2, 'inf'::DOUBLE), log('inf'::DOUBLE, 2), log(2, 'nan'::DOUBLE), log('nan'::DOUBLE, 2)")
run("SELECT log(0)")
run("SELECT log(2, 0)")
run("SELECT log(0, 8)")
run("SELECT log(-1)")
run("SELECT log(-2, 8)")
run("SELECT log(1, 8)")
run("SELECT log(2, 8, 3)")
run(f"SELECT log('abc') FROM {TABLE} LIMIT 1")

[(None, None, None, 0.0, 2.0, 308.0, -323.3062153431158)]
[(inf, nan, 3.0, 0.33333333333333337, 3.0, -3.0, 0.0)]
[(inf, 0.0, nan, nan)]
OSError: cannot take logarithm of zero
OSError: cannot take logarithm of zero
OSError: cannot take logarithm of zero
OSError: cannot take logarithm of a negative number
OSError: cannot take logarithm of a negative number
OSError: divison by zero in based logarithm
InternalError: Binder Error: No function matches the given name and argument types 'log(INTEGER_LITERAL, INTEGER_LITERAL, INTEGER_LITERAL)'. You might need to add explicit type casts.
OSError: Could not convert string 'abc' to DOUBLE

Examples

The base-10 logarithm of each value, its logarithm to the row's own base, and its logarithm to base 2:

run(f"SELECT id, b, x, log(x), log(b, x), log(2, x) FROM {TABLE} ORDER BY id LIMIT 10")

[(1, 2.0, 8.0, 0.9030899869919435, 3.0, 3.0), (2, 10.0, 1000.0, 3.0, 3.0, 9.965784284662087), (3, 0.5, 8.0, 0.9030899869919435, -3.0, 3.0), (4, 3.0, 1.0, 0.0, 0.0, 0.0), (5, None, None, None, None, None)]

Keep only the values of 1,000 or more, by their base-10 logarithm:

run(f"SELECT id, x FROM {TABLE} WHERE log(x) >= 3 ORDER BY id LIMIT 10")

[(2, 1000.0)]

How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (VAST_SCAN). VAST DataBase can evaluate a comparison of a column with a constant, but not a function of a column, so log() of a column is always computed by the Query Engine:

• In the SELECT list, the scan reads the columns and the Query Engine computes log() for each row it returns.
• In WHERE, the scan sends the columns for every row, and a FILTER step in the Query Engine computes log() and keeps the rows that match. No rows are skipped in storage, so every row's value reaches log(): a zero or a negative value fails the query, even in WHERE.

The plan of the WHERE example shows the FILTER step above the scan:

plan(f"SELECT id FROM {TABLE} WHERE log(x) >= 3")

┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│      (log(x) >= 3.0)      │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│             x             │
│             id            │
│                           │
└───────────────────────────┘

Make it faster

• Raise the constant to the power instead. A larger number always has a larger logarithm, so log(x) >= 3 is the same as x >= 1000. That comparison of the raw column with a constant is pushed down to VAST DataBase, so rows that don't match never leave storage, and zero or negative values never reach a logarithm. Its plan has no FILTER step:

plan(f"SELECT id FROM {TABLE} WHERE x >= 1000")

┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘

• Store the logarithm if you filter on it often. Compute it when you write the rows and keep it in its own column.
• Combine it with a pushed-down condition. In id > 100 AND log(x) >= 3, id > 100 is still evaluated in VAST DataBase, so log() runs only on the rows that pass it.

Limitations and errors

• Zero, a negative value or a base of 1 fails the whole query. One such row in a column stops the query for every row. To skip those rows, filter them out with a pushed-down condition such as WHERE x > 0. To keep them in the result with a NULL, use CASE WHEN x > 0 THEN log(x) END.
• One or two arguments, numbers only. log(2, 8, 3) fails, and so does a boolean, date, interval, BLOB or list, with No function matches the given name and argument types. The error lists the two forms log accepts: log(DOUBLE) -> DOUBLE and log(DOUBLE, DOUBLE) -> DOUBLE. A string is converted when the query runs, and one that isn't a number fails with Could not convert string 'abc' to DOUBLE.
• The base comes first. log(8, 2) is the logarithm of 2 to base 8, not of 8 to base 2.

run(f"SELECT id, CASE WHEN x > 0 THEN log(x) END FROM {TABLE} ORDER BY id LIMIT 10")
run("SELECT log(2, 8), log(8, 2)")

[(1, 0.9030899869919435), (2, 3.0), (3, 0.9030899869919435), (4, 0.0), (5, None)]
[(3.0, 0.33333333333333337)]

Similar functions

run("SELECT log(8), log10(8), ln(8), log2(8), log(2, 8), ln(8) / ln(2)")

[(0.9030899869919435, 0.9030899869919435, 2.0794415416798357, 3.0, 3.0, 3.0)]

Clean up:

run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()

[]
[]

See also

• Numeric and math functions: all the numeric functions.
• log10, ln, log2 and exp.
• Where filters run and pushdowns.
• EXPLAIN: how to read a plan.