---
title: "list_entropy"
slug: "dev-database-reference-sql-functions-lists-list-entropy"
updated: 2026-10-30T18:19:06Z
published: 2026-10-30T18:19:06Z
canonical: "docs.vastdata.com/dev-database-reference-sql-functions-lists-list-entropy"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vastdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# list_entropy

*Applies to VAST Cluster 5.5.*

*[Functions](/develop/docs/dev-database-reference-sql-functions-index) › [List, struct and map functions](/develop/docs/dev-database-reference-sql-functions-lists-and-structs) › Aggregate a list.* Returns the Shannon entropy of the values in a list, in bits: how unpredictable an element picked from the list is. It works on lists of any type, and NULL elements are skipped.

## Syntax

```
list_entropy(l)
```

| Argument | Type | Description |
| --- | --- | --- |
| `l` | a list of any type | The values whose entropy is computed. |

**Returns:** `DOUBLE`.

## Set up the examples

The examples connect with the ADBC driver, as in [VAST Query Engine (ADBC)](/develop/docs/dev-database-integrations-adbc), and work in a schema called `docs_sql_fn_lists_list_entropy`. Set `QE_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `VASTDB_BUCKET` ([SQL reference](/develop/docs/dev-database-reference-sql-index#example-connect-and-use-a-transaction)), and run the blocks in order in one Python session. Below each block is the output you should see.

This block connects, defines the `run()` and `plan()` helpers, and creates a table of five rows of lists: whole numbers (`nums`), measurements (`vals`) and words (`words`). Row 2 has NULL elements, row 3 has empty lists, row 4 has no values, and row 5 has non-ASCII words and an empty string:

```
import os
import re

import adbc_driver_manager
import adbc_driver_manager.dbapi as dbapi
import adbc_driver_vastdb

BUCKET = os.environ["VASTDB_BUCKET"]
SCHEMA = f'"{BUCKET}/docs_sql_fn_lists_list_entropy"'

conn = dbapi.connect(
    driver=adbc_driver_vastdb.get_driver_path(),
    db_kwargs={
        "vast.db.endpoint": os.environ["QE_ENDPOINT"],
        "vast.db.access_key": os.environ["AWS_ACCESS_KEY_ID"],
        "vast.db.secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
    autocommit=True,
)

def server_message(e):
    """The server's message in a driver error, without the request ID or the echoed statement."""
    text = re.sub(r"\[req=0x[0-9a-f]+\]: ", "", str(e))
    found = (re.search(r'message: "(.*?)", request_id', text, re.S)
             or re.search(r"description: (.*)\)", text, re.S))
    message = found.group(1) if found else text.strip().splitlines()[-1]
    message = re.sub(r"^Plan query: '.*?' failed with: ", "", message, flags=re.S)
    message = re.split(r"\\n|\n", message)[0].replace('\\"', '"')
    return message.replace(BUCKET, "<bucket>")

def run(sql):
    """Run one statement. Print the rows it returns, or the error."""
    try:
        with conn.cursor() as cur:
            cur.execute(sql)
            rows = cur.fetchall() if cur.description else []
    except (adbc_driver_manager.Error, OSError) as e:
        print(f"{type(e).__name__}: {server_message(e)}")
    else:
        print(rows)

def plan(sql):
    """Print the plan without its ~N Rows lines: the table's row count, which can still be 0 just after an insert."""
    with conn.cursor() as cur:
        cur.execute("EXPLAIN " + sql)
        steps = cur.fetchall()[0][1]
    print("\n".join(line for line in steps.splitlines() if "Rows" not in line))

TABLE = f"{SCHEMA}.samples"
run(f"CREATE SCHEMA IF NOT EXISTS {SCHEMA}")
run(f"CREATE TABLE IF NOT EXISTS {TABLE} (id BIGINT, nums BIGINT[], vals DOUBLE[], words VARCHAR[])")
run(f"INSERT INTO {TABLE} VALUES "
    "(1, [1, 2, 3], [0.5, 2.5, -1.0], ['a', 'b', 'c']), "
    "(2, [3, NULL, 3], [2.0, NULL], ['b', NULL, 'B']), "
    "(3, [], [], []), "
    "(4, NULL, NULL, NULL), "
    "(5, [-5, 0, -5, 10], [1e308, 1e308, -1e-308], ['\u00e9', 'Zo\u00eb', '\U0001F1EB\U0001F1F7', ''])")
```

```
[]
[]
[(5,)]
```

## What it does

`list_entropy(l)` counts how often each distinct non-NULL value occurs in `l`, turns the counts into proportions `p`, and returns the sum of `-p &#xd7; log2(p)`:

- One value repeated gives `0.0`: `[1, 1, 1]`.
- Two values, equally often, give `1.0` (one bit): `[1, 1, 2, 2]`. Four values, equally often, give `2.0`: `[1, 2, 3, 4]`.
- Uneven counts give less than equal ones: `[&apos;a&apos;, &apos;a&apos;, &apos;b&apos;]` gives `0.9182958340544893`.
- Only how often values occur matters, not their size or order, so it works on text and other types as well as numbers. Text is compared exactly: `&apos;b&apos;` and `&apos;B&apos;` are different values.

The counts are those of [list_histogram](/develop/docs/dev-database-reference-sql-functions-lists-list-histogram).

## NULLs and edge cases

| Input | Result |
| --- | --- |
| `NULL`, or `NULL::BIGINT[]` | `NULL` |
| `[]::BIGINT[]` (empty list) | `0.0`, not NULL |
| `[NULL, 1]` | `0.0` (the NULL is skipped) |
| `[1, 1, 1]` | `0.0` |
| `[&apos;a&apos;, &apos;a&apos;, &apos;b&apos;]` | `0.9182958340544893` |
| `[&apos;NaN&apos;::DOUBLE, &apos;NaN&apos;::DOUBLE]` | `1.0`: each `NaN` counts as a different value |
| `typeof(list_entropy([1, 2]))` | `&apos;DOUBLE&apos;` |
| A number, string, `BLOB`, date or boolean | Error: `No function matches the given name and argument types`, for example `&apos;list_aggr(INTEGER_LITERAL, STRING_LITERAL)&apos;` |

The same cases, as the cluster returns them. They use the table and helpers from [Set up the examples](/develop/docs/dev-database-reference-sql-functions-lists-list-entropy#set-up-the-examples):

```
run("SELECT list_entropy(NULL), list_entropy(NULL::BIGINT[]), list_entropy([]::BIGINT[]), list_entropy([NULL, 1])")
run("SELECT list_entropy([1, 1, 1]), list_entropy(['a', 'a', 'b']), list_entropy(['NaN'::DOUBLE, 'NaN'::DOUBLE]), "
    "typeof(list_entropy([1, 2]))")
run(f"SELECT list_entropy(123) FROM {TABLE} LIMIT 1")
```

```
[(None, None, 0.0, 0.0)]
[(0.0, 0.9182958340544893, 1.0, 'DOUBLE')]
InternalError: Binder Error: No function matches the given name and argument types 'list_aggr(INTEGER_LITERAL, STRING_LITERAL)'. You might need to add explicit type casts.
```

## Examples

The entropy of each row's lists. Row 1's three different values give `1.5849625007211559` (log2 of 3) in every column. Row 2's words `[&apos;b&apos;, NULL, &apos;B&apos;]` give `1.0`: the NULL is skipped and `&apos;b&apos;` and `&apos;B&apos;` differ. Row 3's empty lists give `0.0`, and row 4's NULL lists give NULL:

```
run(f"SELECT id, list_entropy(nums), list_entropy(vals), list_entropy(words) FROM {TABLE} ORDER BY id LIMIT 10")
```

```
[(1, 1.5849625007211559, 1.5849625007211559, 1.5849625007211559), (2, 0.0, 0.0, 1.0), (3, 0.0, 0.0, 0.0), (4, None, None, None), (5, 1.5, 0.9182958340544893, 2.0)]
```

The rows whose words are more varied than one bit:

```
run(f"SELECT id, words FROM {TABLE} WHERE list_entropy(words) > 1 ORDER BY id LIMIT 10")
```

```
[(1, ['a', 'b', 'c']), (5, ['é', 'Zoë', '🇫🇷', ''])]
```

## How it runs in VAST

The Query Engine runs on the cluster's CNodes, and a query on a table starts with a scan in VAST DataBase (`VAST_SCAN`). The scan reads a list column whole, and `list_entropy()` always runs in the Query Engine:

- **In the `SELECT` list,** a projection step computes `list_entropy()` for each row the scan returns.
- **In `WHERE`,** the scan sends the list for every row, and a `FILTER` step computes `list_entropy()` and keeps the rows that match. No rows are skipped in storage.

The plan of the `WHERE` example shows the `FILTER` step above the scan:

```
plan(f"SELECT id FROM {TABLE} WHERE list_entropy(words) > 1")
```

```
┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #1            │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│ (list_aggr(words, 'entropy│
│         ') > 1.0)         │
│                           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│                           │
│        Projections:       │
│           words           │
│             id            │
│                           │
└───────────────────────────┘
```

## Make it faster

- **Add a condition on a plain column.** A comparison of a column with a constant, such as `id &gt; 1`, is evaluated in VAST DataBase, so `list_entropy()` runs only on the rows that pass it. The plan for `id &gt; 1` alone has no `FILTER` step:

```
plan(f"SELECT id FROM {TABLE} WHERE id > 1")
```

```
┌───────────────────────────┐
│         VAST_SCAN         │
│    ────────────────────   │
│    Function: VAST_SCAN    │
│      Projections: id      │
│                           │
└───────────────────────────┘
```

- **Store the entropy if you filter on it often.** Compute it when you write the rows and keep it in its own column: `entropy &gt; 1` is pushed down, `list_entropy(words) &gt; 1` isn't.

## Limitations and errors

- **An empty list gives `0.0`, not NULL.** Test for empty lists separately if you need to tell them apart from lists of one repeated value, for example with `len(l) = 0`.
- **`NaN` values aren't grouped.** Each `NaN` counts as a value of its own.
- **Lists only.** Any other type fails with `No function matches the given name and argument types`. The error names `list_aggr`, the list aggregate that `list_entropy` runs on, and its accepted form `list_aggr(ANY[], VARCHAR, [ANY...]) -&gt; ANY`.
- **`ORDER BY` needs a `LIMIT`.** Without one, the query fails with `The operator &apos;ORDER_BY&apos; type=1 is currently not supported for this usage`.

## Similar functions

| Function | What it returns | Result for `[3, 1, NULL, 3, 10]` |
| --- | --- | --- |
| `list_entropy` | entropy of the value counts, in bits | `1.5` |
| [list_histogram](/develop/docs/dev-database-reference-sql-functions-lists-list-histogram) | the count of each distinct value, as a map | `[(1, 1), (3, 2), (10, 1)]` (the driver shows the map as key and count pairs) |
| [list_unique](/develop/docs/dev-database-reference-sql-functions-lists-list-unique) | the number of distinct non-NULL values | `3` |
| [list_approx_count_distinct](/develop/docs/dev-database-reference-sql-functions-lists-list-approx-count-distinct) | an estimate of the number of distinct values | `3` |

```
run("SELECT list_entropy(l), list_histogram(l), list_unique(l), list_approx_count_distinct(l) "
    "FROM (SELECT [3, 1, NULL, 3, 10] AS l)")
```

```
[(1.5, [(1, 1), (3, 2), (10, 1)], 3, 3)]
```

Clean up:

```
run(f"DROP TABLE IF EXISTS {TABLE}")
run(f"DROP SCHEMA IF EXISTS {SCHEMA}")
conn.close()
```

```
[]
[]
```

## See also

- [List, struct and map functions](/develop/docs/dev-database-reference-sql-functions-lists-and-structs): all the list functions.
- [list_histogram](/develop/docs/dev-database-reference-sql-functions-lists-list-histogram): how often each value occurs.
- [Where filters run](/develop/docs/dev-database-reference-sql-select#where-filters-run) and [pushdowns](/develop/docs/dev-database-reference-pushdowns).
- [EXPLAIN](/develop/docs/dev-database-reference-sql-explain): how to read a plan.