Iceberg Hilbert Curve: Faster Multi-Column Queries Than Z-Order

Your table can only be sorted one way. Your users filter it ten different ways. That tension has shaped data lake design for years: sort by date and customer lookups crawl; sort by customer and every dashboard scans the whole table. Most teams quietly live with it, or keep two copies of the same data.

Apache Iceberg just got a better answer: the Iceberg Hilbert curve sort order. Its standard compaction procedure, rewrite_data_files, can now arrange rows along a Hilbert curve, a space-filling curve that keeps rows close on several columns at once, and does it more cleanly than the Z-order option Iceberg already had. The change itself is one new word in your SQL (source). What it unlocks is much bigger.

This one is personal for me. I’ve been following Iceberg for a few years now, and in May 2025 I wrote that Iceberg isn’t just a table format, it’s a philosophy shift: open, engine-agnostic, and smart enough to skip data using its own metadata. Hilbert clustering is that last idea taken one step further, and it closes a gap I’d been quietly waiting on.

In this post I’ll explain the ideas behind it from the ground up: how queries skip files, why Z-order was a good first step, what makes a Hilbert curve better, and what becomes possible now that it lives inside Iceberg. I also ran a small simulation with real numbers, so you can see the difference rather than take my word for it.

The problem: one table, many query patterns

Picture an events table with a few billion rows. The analytics team filters it by date for dashboards. The support team looks up one customer’s history. The product team asks for one region over a week. Same table, three access patterns, and each one wants the data arranged differently.

How a query skips files: min/max pruning (data skipping)

File pruning (also called data skipping) is how a query engine avoids opening data files that can’t contain matching rows, using per-file min/max statistics. A big Iceberg table is really thousands of Parquet files, and Iceberg’s metadata records the minimum and maximum value of each column in every file. When you run WHERE event_date BETWEEN '2026-09-10' AND '2026-09-12', the engine checks those ranges first and skips every file that can’t contain those dates, without opening it. This is file pruning, and it’s the single biggest lever for fast, cheap queries on large tables: fewer files opened means less I/O, less shuffle and a smaller compute bill.

Pruning only works if file ranges are narrow. If every file holds rows from the whole month, every file’s range covers your three days, and nothing gets skipped. Two details are worth knowing. By default Iceberg keeps these ranges for the first 100 columns and truncates string ranges to 16 characters. And inside each Parquet file, row groups carry their own min/max too, so good clustering pays off twice.

Why one sort order isn’t enough

The classic fix is to sort. Sort by event_date and each file holds a thin slice of dates, so date filters prune beautifully. But the customer lookup and the region query now hit files that each contain every customer and every region. In the simulation later in this post, a table sorted by two columns had to open all 64 files for a filter on the second column alone.

Hence the old workaround: keep a second copy of the table sorted differently, and pay for the storage, the pipelines and the “which copy is right?” conversations. What we really want is a single arrangement that keeps files narrow on several columns at once. That’s what space-filling curves do.

Z-order clustering in Iceberg: the first answer, and its crack

How bit interleaving works

Z-order clustering sorts rows by a key made from interleaving the bits of several columns, so rows close on all of those columns tend to land in the same files. Iceberg has supported zorder(...) for years. The trick is elegant: write each column’s value in binary, then interleave the bits, one from x, one from y, one from x, and so on. Sorting rows by that combined number keeps rows that are close in both columns mostly close on disk. Iceberg first converts each value (a date, a string, a number) into an order-preserving byte form, so it works across column types.

The crack: occasional big jumps

Bit interleaving isn’t smooth. Walk the grid in Z-order and most steps are short, but at the boundary between big blocks the path leaps from one side of the space to the other. On the 8×8 grid above, the longest Z-order step spans 8 cells, and 49% of steps don’t land on a touching cell. On a 1024×1024 grid the longest step spans 1,024 cells.

Why does that matter? Files are written by cutting the sorted rows into chunks. When a chunk contains one of those jumps, it gets rows from two distant regions, and its min/max range stretches to cover both. Wider ranges mean more files overlap your filter, so more files get opened.

What is a Hilbert curve? The same idea, with no jumps

How a Hilbert curve is built

A Hilbert curve is a space-filling curve: a single continuous line that visits every cell of a grid exactly once, always stepping to an adjacent cell. Sorting rows by their position along it keeps rows that are close on several columns close together on disk.

Described by David Hilbert in 1891, it’s a single line folded so tightly that it passes through every cell of a grid, never crossing itself and never jumping. You build it recursively: start with a U through four cells, then replace each cell with a smaller U, rotated so the pieces join end to end, and repeat. Each row’s position along that line becomes its sort key, in place of Z-order’s interleaved bits. The same construction works for three, four or more columns.

If you like to see the mechanics, here’s the classic two-column version in Python:

def hilbert_index(x: int, y: int, bits: int) -> int:
    """Position of cell (x, y) along a Hilbert curve on a 2^bits x 2^bits grid."""
    n = 1 << bits
    d, s = 0, n >> 1
    while s > 0:
        rx = 1 if x & s else 0
        ry = 1 if y & s else 0
        d += s * s * ((3 * rx) ^ ry)
        if ry == 0:                      # rotate the quadrant so the pieces join up
            if rx == 1:
                x, y = n - 1 - x, n - 1 - y
            x, y = y, x
        s >>= 1
    return d

# 2x2 grid: (0,0) -> 0, (0,1) -> 1, (1,1) -> 2, (1,0) -> 3  (a U shape)

What “no jumps” does and doesn’t guarantee

It’s easy to oversell this, so let’s be precise:

• Guaranteed: rows that are next to each other in the sort order are neighbours in the data space. A file is a contiguous run of sorted rows, so every file stays in one compact region.
• Not guaranteed: that every pair of neighbours in the data space ends up close in the sort order. No curve can do that; a few boundaries always split close points apart.

For pruning, the first property is the one that counts, and it’s why Hilbert-clustered files have tighter ranges.

Hilbert vs Z-order: a simulation with real numbers

Diagrams are nice; numbers are better. I wrote a short Python simulation of exactly this mechanism. It isn’t a benchmark of Iceberg itself; it isolates the idea.

• Data: 1,000,000 rows with two filter columns, x and y, each on a 0–1023 scale. 70% are spread evenly and 30% sit in six hot spots, like busy dates or big customers.
• Layouts: the rows are sorted three ways (plain ORDER BY x, y, Z-order and Hilbert), then cut into 64 equal files, keeping each file’s min and max per column, the same statistics Iceberg keeps.
• Queries: 2,000 random range filters per workload: small and large boxes on both columns, x only, and y only. A file counts as opened if its ranges overlap the filter.

Here is the code, one mechanism per block. Each one is just a key function: sort the rows by it and you get that layout. Paste all five blocks, in order, into one file and it reproduces the results table below in about a second (it only needs NumPy).

import numpy as np

BITS = 10                      # each column scaled to 0..1023
N_ROWS, N_FILES = 1_000_000, 64
rng = np.random.default_rng(42)

# 70% of rows spread evenly, 30% in six hot spots
uniform = rng.integers(0, 1 << BITS, size=(700_000, 2))
centers = rng.integers(100, 924, size=(6, 2))
hot = np.concatenate([np.clip(rng.normal(c, 40, size=(50_000, 2)), 0, (1 << BITS) - 1)
                      for c in centers]).astype(np.int64)
data = np.concatenate([uniform, hot])
X, Y = data[:, 0].astype(np.int64), data[:, 1].astype(np.int64)

def linear_key(x, y):
    """ORDER BY x, y: x decides everything, y only breaks ties."""
    return x * (1 << BITS) + y

def zorder_key(x, y):
    """Z-order (Morton): interleave the bits of x and y -> x9 y9 x8 y8 ... x0 y0."""
    z = np.zeros_like(x, dtype=np.int64)
    for b in range(BITS):
        z |= ((x >> b) & 1) << (2 * b + 1)   # x bit goes to the odd position
        z |= ((y >> b) & 1) << (2 * b)       # y bit goes to the even position
    return z

def hilbert_key(x, y):
    """Hilbert curve position (classic xy -> d), vectorised with NumPy.
    At each level: pick the quadrant, add how far along the curve it is,
    then rotate/flip the point so the smaller curve inside it lines up."""
    x, y = x.astype(np.int64).copy(), y.astype(np.int64).copy()
    d = np.zeros_like(x)
    s = (1 << BITS) >> 1
    while s > 0:
        rx = ((x & s) > 0).astype(np.int64)
        ry = ((y & s) > 0).astype(np.int64)
        d += s * s * ((3 * rx) ^ ry)          # quadrant order: 0, 1, 2, 3 along the U
        m = ry == 0                           # bottom quadrants need a rotation
        flip = m & (rx == 1)
        x[flip] = s - 1 - x[flip]
        y[flip] = s - 1 - y[flip]
        x[m], y[m] = y[m].copy(), x[m].copy() # swap x and y
        s >>= 1
    return d

def file_stats(key):
    """Sort rows by the key, cut into equal files, keep min/max per column (like Iceberg)."""
    order = np.argsort(key, kind="stable")
    xs, ys = X[order], Y[order]
    return np.array([[xs[c].min(), xs[c].max(), ys[c].min(), ys[c].max()]
                     for c in np.array_split(np.arange(N_ROWS), N_FILES)])

def random_filters(n, frac_x, frac_y):
    """n random range filters: WHERE x BETWEEN .. AND y BETWEEN .."""
    wx, wy = int(frac_x * (1 << BITS)), int(frac_y * (1 << BITS))
    x0 = rng.integers(0, (1 << BITS) - wx + 1, n)
    y0 = rng.integers(0, (1 << BITS) - wy + 1, n)
    return np.stack([x0, x0 + wx - 1, y0, y0 + wy - 1], axis=1)

def files_opened(st, q):
    """A file is opened only if its [min, max] overlaps the filter on BOTH columns."""
    hit = ((st[None, :, 0] <= q[:, None, 1]) & (st[None, :, 1] >= q[:, None, 0]) &
           (st[None, :, 2] <= q[:, None, 3]) & (st[None, :, 3] >= q[:, None, 2]))
    return hit.sum(axis=1).mean()

layouts = {"ORDER BY x, y": file_stats(linear_key(X, Y)),
           "Z-order":       file_stats(zorder_key(X, Y)),
           "Hilbert":       file_stats(hilbert_key(X, Y))}
workloads = {"x and y, 5% box":  random_filters(2000, 0.05, 0.05),
             "x and y, 20% box": random_filters(2000, 0.20, 0.20),
             "x only, 5%":       random_filters(2000, 0.05, 1.0),
             "y only, 5%":       random_filters(2000, 1.0, 0.05)}

print(f"{'files opened (of 64)':<20}" + "".join(f"{name:>15}" for name in layouts))
for wname, q in workloads.items():
    print(f"{wname:<20}" + "".join(f"{files_opened(st, q):>15.1f}" for st in layouts.values()))

The picture explains the results before you see them. A plain sort produces thin, full-height slices: perfect for x, useless for y. Z-order produces blocks that serve both columns, but files that straddle a jump cover more space. Hilbert’s files are compact on both columns. Across 64 files, the average file covered 2.2% of the space with Hilbert versus 4.2% with Z-order.

Average files opened per query, out of 64. I re-ran it with two other random seeds and the numbers moved by less than 0.6 files.

Three lessons:

1. Hilbert beat Z-order on every workload, by 25–39% on two-column filters.
2. A plain sort is still best if you only ever filter on one column. Multi-column clustering is a trade: each column gives up a little so that all columns do well.
3. The worst case is what hurts. The plain sort opened all 64 files for a y filter. Both curves avoid that cliff; Hilbert does it with tighter files.

Real tables will differ: more columns, strings and timestamps, skewed data. Treat these as directional numbers and measure on your own table: cluster one copy with zorder and one with hilbert, then compare scanned files for your ten most common queries.

Why Hilbert clustering in Apache Iceberg is exciting

Why I’ve been waiting for this

The first time I looked at Iceberg seriously, I was fighting append-only Snowpipe pipelines, with late-arriving data, reprocessing and duplicate compute. Iceberg promised a way out, but support was still early, so I stayed with what worked. I’ve watched it close one gap after another since then.

When I wrote my Iceberg deep-dive in 2025, I made two arguments that this change directly builds on:

• Metadata-driven pruning is Iceberg’s quiet superpower. I wrote that Iceberg “leverages manifest files and metadata layers to prune irrelevant data before reading.” Hilbert clustering makes exactly those per-file statistics sharper, so the same metadata now skips far more files.
• Partition evolution is the hidden superpower for layout. Partitions decide which part of a table a query touches. Clustering is the missing other half: how tightly rows are packed inside those partitions. Together they cover both levels of the layout.

My comparison of Delta, Hudi and Iceberg in that post focused on transactions, schema and partition evolution, and engine support. Honestly, data layout optimization was the one area where Iceberg was behind: Hudi and Delta already clustered with Hilbert curves, and Iceberg users had Z-order. That’s the gap this closes.

Three reasons it matters

Hilbert clustering isn’t new to open source: Apache Hudi has had it since 2021, and Delta Lake’s liquid clustering is built on it. What’s new is the Iceberg Hilbert curve option: native to the table format, and that matters more than it sounds:

1. Iceberg is the format the most engines share. Spark, Trino, Flink, Snowflake, Athena, BigQuery, Dremio and others read Iceberg tables, and many write them too. The benefit of clustering lives in the table’s file statistics, so you cluster once and every engine that reads the table skips the same files. You don’t need a specific engine or vendor to get the speed-up.
2. No format switch. Until now, Iceberg teams who wanted Hilbert-quality clustering had to switch table formats or rely on a vendor platform’s own optimizer. Now it’s one word in the compaction procedure they already run.
3. It arrives with geospatial types. Iceberg 1.12 also wires through the geometry and geography types added in the V3 spec. Clustering on coordinates is what makes spatial filters prune well (more on that below).

What becomes possible with Hilbert clustering

One table for many query patterns

Back to our events table, and the most immediate win of the Iceberg Hilbert curve. Instead of choosing between date and customer, or keeping two copies, you cluster on both: hilbert(event_date, customer_id). Dashboards filtering by date, support looking up a customer, and analysts asking for one customer over one month all prune well from the same table. Fewer copies means less storage, fewer pipelines to keep in sync, and one source of truth.

A real geospatial lakehouse

Dates and IDs at least have an order. Map coordinates don’t: there’s no meaningful answer to “is Paris bigger than Tokyo?”. Sort a trips table by longitude and cities at the same longitude but different latitudes end up in the same files, so spatial filters prune poorly.

A Hilbert curve gives the map a locality-preserving order: nearby points get nearby positions, so each file becomes something like a compact map tile. Think of the trips table from my ride-hailing data model post. Clustered on pickup latitude and longitude (plus time), “all pickups in this neighbourhood last Friday” opens a handful of tiles instead of the whole city. Delivery zones, fleet telemetry, store footfall and IoT sensors all have the same shape of question. Combined with Iceberg’s new geometry types, this is what makes geospatial analytics on an open lakehouse practical.

Fast entity-plus-time lookups

“Show me everything this customer, device or account did between these dates” is one of the most common questions in data engineering. It powers customer 360 views, support tools, fraud investigations and feature tables for ML. With hilbert(entity_id, event_time), both halves of the filter prune, so these lookups become fast without a separate index or a copy sorted by entity.

Cheaper, snappier dashboards

BI dashboards are multi-column filters by nature: date, region, product, channel. Every file a dashboard query doesn’t open is money you don’t spend and seconds your users don’t wait. Tighter files also mean fewer rows decoded once a file is opened, because Parquet row groups inside it are clustered too (see why Parquet beats CSV on cost).

Clustering that travels between engines

Today the clustering choice lives in your compaction job: you pass 'hilbert(c1, c2)' when you run it. The Iceberg community is now discussing the next step for the V4 table spec: storing the clustering definition itself in table metadata, with every parameter spelled out, for example per-column scaling so mixed types use the curve well. If that lands, any engine could read the table’s clustering intent and keep the layout up to date, not just the job that wrote it. That’s the path toward automatic, “liquid” clustering in a fully open format. It’s still under discussion, but it’s the right direction.

How to use the Iceberg Hilbert curve in rewrite_data_files

Once you’re on Iceberg 1.12 with Spark 4.1 (Spark 4.0 support is planned), switching to the Iceberg Hilbert curve is one word in the rewrite_data_files procedure:

-- what you may run today
CALL catalog.system.rewrite_data_files(
  table      => 'db.events',
  strategy   => 'sort',
  sort_order => 'zorder(event_date, customer_id)'
);

-- with Hilbert clustering
CALL catalog.system.rewrite_data_files(
  table      => 'db.events',
  strategy   => 'sort',
  sort_order => 'hilbert(event_date, customer_id)',
  options    => map('rewrite-all', 'true')   -- first run: re-cluster every file
);

Practical tips:

• Pick 2–4 columns that your queries filter on most. Every extra column dilutes the clustering of the others.
• Re-run on a schedule. New data arrives unclustered, so run compaction nightly or weekly on recent partitions.
• Filter on the raw clustered columns. Pruning uses each column’s stored min/max, so WHERE event_date = … prunes, while filters on expressions the engine can’t map back to the column may not.
• Expect a little more CPU at write time. Computing a Hilbert position costs slightly more than interleaving bits. You pay it once during compaction, and every read afterwards benefits.
• Watch long strings. Iceberg’s curve encodings only look at the first few bytes of variable-length values (var-length-contribution, default 8 for Z-order), so strings with long common prefixes, such as URLs, won’t cluster well with either curve.
• Measure before you switch. Try it on a copy or a few partitions and compare scanned files for your common queries.

Summary: Hilbert vs Z-order in Apache Iceberg

Key takeaways

• Fast queries on big tables come from skipping files, and that needs each file to cover a narrow range of values.
• A plain sort helps one column and can be useless for the others (64 of 64 files in my simulation).
• Z-order handles several columns, but its occasional big jumps widen file ranges.
• The Iceberg Hilbert curve never jumps, so files stay compact on every clustered column: 25–39% fewer files opened than Z-order for two-column filters in my simulation.
• Having it inside Iceberg is the real news: cluster once, and every engine that reads the table benefits, with no format switch.
• It unlocks new designs: one table instead of many copies, geospatial analytics on an open lakehouse, fast entity-plus-time lookups and cheaper dashboards.

Cheat sheet: plain sort vs Z-order vs Hilbert

The part that excites me

For years, “sort your table” meant “pick a winner”. The Iceberg Hilbert curve quietly retires that trade-off for most tables, and it does so in the open format every major engine already speaks. The idea is from 1891. Having it one word away, in a table format anyone can use, is what’s new.

In May 2025 I ended my Iceberg post with a slightly provocative line: “If you’re not already using Iceberg — you’re building on yesterday’s architecture.” Sixteen months later, I’d keep it and add one more: if you’re already on Iceberg, it’s time to look at how your data is laid out, not just where it’s stored. Hilbert clustering is the easiest place to start.

Are you clustering with zorder today? Tell me in the comments which columns you cluster on and which query patterns you’d like to speed up. If you test hilbert once 1.12 lands, I’d love to see your scanned-files numbers.

Frequently asked questions

What is Iceberg Hilbert curve clustering?

Iceberg Hilbert curve clustering is a sort order for Iceberg’s rewrite_data_files procedure that arranges rows by their position along a Hilbert curve across several columns. Rows that are close on all of those columns end up in the same data files, so each file covers a narrow range and queries can skip more files.

How is a Hilbert curve different from Z-order?

Both combine several columns into one sort key. Z-order interleaves the bits of each column, which occasionally makes big jumps across the data space and widens file ranges. A Hilbert curve never jumps: consecutive positions are always neighbouring cells, so files stay compact on every clustered column.

Why is Hilbert clustering in Iceberg a big deal?

Iceberg is the table format the most query engines share. Clustering improves the table’s own file statistics, so you cluster once and Spark, Trino, Flink, Snowflake, Athena and other engines that read the table all skip the same files, without switching formats or vendors.

Which Iceberg version supports hilbert() in rewrite_data_files?

It is expected in Apache Iceberg 1.12, starting with Spark 4.1, with Spark 4.0 support planned. Until then, zorder() is the multi-column option.

How do I use Hilbert clustering in Iceberg?

Call rewrite_data_files with strategy ‘sort’ and sort_order ‘hilbert(col1, col2)’, exactly as you would with ‘zorder(col1, col2)’. Pick the 2 to 4 columns your queries filter on most, and re-run compaction regularly as new data arrives.

Is Hilbert always better than Z-order?

For multi-column range filters it usually prunes better; in our simulation it opened 25 to 39 percent fewer files than Z-order. But if nearly every query filters on a single column, a plain sort on that column is still best. Measure on your own tables before switching.

Does Hilbert clustering help geospatial queries?

Yes, that is one of its biggest use cases. Coordinates have no natural one-dimensional order, and a Hilbert curve gives nearby points nearby positions, so files behave like compact map tiles that spatial filters can skip.

Does clustering replace partitioning in Iceberg?

No. Partitioning decides which parts of the table a query plans over, while clustering decides how tightly rows are packed within them. A common pattern is to partition coarsely by time and then Hilbert-cluster inside each partition on the columns you filter by.

What is file pruning (data skipping) in Iceberg?

Iceberg stores the minimum and maximum value of each column for every data file. When a query filters on a column, the engine skips every file whose range cannot contain matching values, without opening it. Clustering with Z-order or Hilbert curves makes those ranges narrower, so more files are skipped.

Further reading on DataForGeeks:

• Apache Iceberg: the data lake breakthrough reshaping big data
• One Ride, 400 Milliseconds: a ride-hailing data model
• How to debug a slow Spark job and fix data skew
• Cost and performance analysis: CSV vs Parquet
• Apache Spark performance tuning and best practices