Vector search on your data lake: Lance and Iceberg, without copying data

Vector search on a data lake means indexing the embeddings that already sit in Lance datasets or Apache Iceberg tables and serving queries over them, without exporting a second copy. Milvus does this with external collections (Milvus 3.0+), which make it zero-copy vector search: you map lake columns to a schema, Milvus builds its own indexes, and search, filtering, and hybrid search run through the standard API while the rows stay in object storage.

Create an external collection Try it on Zilliz Cloud

Milvus 3.0+
Lance, Iceberg, Parquet, Vortex
Source data stays read-only
Apache-2.0, 40K+ GitHub stars

Spark, Ray, or any lake writer owns the rows. Milvus owns only the mapping, manifests, and indexes.

What is a Milvus external collection?

A Milvus external collection is a collection whose rows live in your data lake instead of in Milvus. You set external_source to a Lance dataset, an Iceberg table, or a Parquet or Vortex prefix, and each field's external_field names the source column it reads. Milvus treats the source as read-only and never rewrites or duplicates the files.

The lake stays the single source of truth. Milvus adds the low-latency serving layer on top: ANN indexes, scalar filters, BM25 full-text search, and hybrid search.

In your lake: the authoritative rows, written by Spark, Ray, DuckDB, or training jobs.
In Milvus: schema mapping, manifests, vector, scalar, JSON, and BM25 indexes, and caches.
Same API as native collections: search, query, get, and hybrid search.
No ETL pipeline and no second copy of embeddings to keep in sync.

Do I need a separate vector database if I have a lakehouse?

Not when the embeddings already live in a lakehouse table. The classic pattern exports them into a separate vector database, which doubles storage and adds a sync job for every insert, delete, and schema change. An external collection indexes the table in place. Import remains the right path when the application itself writes vectors through Milvus.

Data lake vs vector database: import vs. external collection
	Import into a native collection	External collection
Where rows live	Copied into Milvus storage	Stay in Lance / Iceberg / Parquet / Vortex
Copies of the data	Two (lake + Milvus)	One (lake)
Who writes	Applications via insert, upsert, delete	Lake jobs (Spark, Ray, Flink, dbt)
Freshness	Visibility follows the selected consistency level	Visible after refresh_external_collection
Indexes	Vector, scalar, JSON, full-text	Vector, scalar, JSON, BM25 over mapped fields
Governance	Split across two systems	Stays with the lake table

What are the ways to run vector search on a lakehouse?

Lakehouse vector search comes in three shapes that differ in where the index lives and who serves the query. Milvus takes the third: a serving layer that indexes the table in place.

Lakehouse vector search approaches: where the vector index lives and what each fits
Approach	Where the index lives	Examples	Fits when
Vector column plus SQL functions in the table format	In the table's own services and query engine	Apache Hudi VECTOR type with hudi_vector_search	SQL-first teams running batch top-k beside joins
Index files owned by the table or file format	Next to the data files, built by the format's tooling	Lance datasets with native IVF_PQ or IVF_HNSW indexes read by LanceDB or DuckDB; Paimon 2.0 vector index read by Doris or StarRocks	Analytics engines that already read the table, in batches
Serving layer that indexes the table in place	In the serving system; the table stays read-only	Milvus external collection over Lance, Iceberg, Parquet or Vortex	Online retrieval with filters, BM25 and hybrid search

Lance vector search: how do you serve a Lance dataset with Milvus?

Lance is an open columnar format for AI data with random row access and versioned datasets. Milvus maps existing Lance columns and fetches requested output fields after retrieval. Point external_source at the dataset directory and set format to lance-table.

Lance vs Parquet comes down to access pattern: Parquet is tuned for sequential scans, while Lance also serves row lookups by position, which suits post-search fetches and training samplers. Milvus reads both without conversion.

Write and append with Lance's Python, Spark, or Ray integrations as usual.
Milvus maps the dataset's fragments into a manifest and builds its own HNSW, IVF or DiskANN index in Milvus storage; it never writes into the dataset, so Lance's native indexes and versions are untouched.
After new fragments land, refresh: unchanged data keeps its segments and indexes.
Store images, text, and embeddings in one table, the multimodal lakehouse pattern; Milvus maps only the columns you need.

Iceberg vector search: how do you index Apache Iceberg embeddings?

Apache Iceberg has no vector type, so embeddings are typically stored as a list or fixed-size list of floats; Milvus reads either as a FLOAT_VECTOR when the length matches dim. Point external_source at the table's metadata JSON file and pin a snapshot_id, so search serves one well-defined table state until you move it forward.

Keep writing the table from Spark, Flink, or any Iceberg engine.
Milvus plans the snapshot's data files and honors position deletes.
Rewrite equality deletes to position deletes before refreshing, for example with compaction.
Advance by refreshing with a newer metadata file and snapshot_id.

How do indexes and refresh keep the lake as the source of truth?

Indexes on lake data work as on native collections; only freshness differs. Refresh is explicit and asynchronous: it scans source metadata and publishes a new manifest, typically in 150–250 ms before index work. Queries keep serving the previous state until the refresh completes, so readers never see a half-applied update.

External fields can also feed Functions, so Milvus generates BM25 sparse vectors or text embeddings from a lake text column without writing a new column back to the table.

Lifecycle of lakehouse vector search with an external collection
Step	What Milvus does	Effect on the lake
Create	Stores the schema and external_field mapping	None
Refresh	Scans metadata, writes a manifest, reuses unchanged segments	Read-only
Index	Builds HNSW/AUTOINDEX, scalar, JSON, and BM25 indexes	None; indexes live in Milvus storage
Serve	Search, query, get, hybrid search; fetches output fields on demand	Reads only
Evolve	add_collection_field maps a new source column, then refresh	None

Can you run vector search on object storage, and what does it cost?

Storage cost is the object storage you already pay for, plus Milvus's manifests and indexes; there is no second copy of the raw rows or the embeddings. Compute is separate from storage, so you size QueryNodes for query load, not for data volume.

One copy of the data in open formats on S3, GCS, Azure, or S3-compatible storage.
Tiered Storage lazy-loads segments and warms up hot fields, so caches can stay smaller than the data.
Read IOPS against the lake are rate-limited and configurable, protecting shared buckets.
Several Milvus instances can serve the same lake dataset at once.

Vector search on the data lake stack: how does Milvus fit with Spark, Ray, and DuckDB?

A vector lakehouse splits the work by strength: batch engines write and transform, analytics engines scan, and Milvus serves low-latency retrieval over the same tables.

Data lake vector search stack with Lance and Iceberg
Layer	Typical tools	Role
Write and embed	Spark, Ray, Flink, PyArrow	Produce rows and embeddings into Lance or Iceberg
Store	Lance datasets, Iceberg tables on object storage	Single source of truth, versioned
Analyze	DuckDB, Spark SQL, Trino	Scans, joins, and evaluation over the same files
Serve	Milvus external collection	ANN, filtered, full-text, and hybrid search
Feed back	Milvus Snapshot, Spark-Milvus connector	Batch jobs read Milvus collections

How do you search a Lance dataset or Iceberg table with Milvus?

Enable Storage V3 (common.storage.useLoonFFI: true), then create a schema that points at the lake source, map fields with external_field, refresh, index, load, and search.

python

import json, os, time
from pymilvus import DataType, MilvusClient

client = MilvusClient(uri="http://localhost:19530", token="root:Milvus")
extfs = {
    "cloud_provider": "aws",
    "region": "us-east-1",
    "access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
    "access_key_value": os.environ["AWS_SECRET_ACCESS_KEY"],
}

def lake_collection(name, source, spec):
    schema = client.create_schema(external_source=source, external_spec=json.dumps(spec))
    # Map Milvus fields to existing source columns.
    schema.add_field(field_name="id", datatype=DataType.INT64, external_field="doc_id")
    schema.add_field(field_name="emb", datatype=DataType.FLOAT_VECTOR, dim=1024, external_field="embedding")
    schema.add_field(field_name="title", datatype=DataType.VARCHAR, max_length=1024, external_field="title")
    client.create_collection(collection_name=name, schema=schema)

# Lance: point at the dataset directory.
lake_collection("docs_lance", "s3://lake/docs.lance",
                {"format": "lance-table", "extfs": extfs})

# Iceberg: point at the table metadata file and pin a snapshot.
lake_collection("docs_iceberg", "s3://lake/docs/metadata/v1.metadata.json",
                {"format": "iceberg-table", "snapshot_id": 123456789, "extfs": extfs})

name = "docs_iceberg"
job_id = client.refresh_external_collection(collection_name=name)
deadline = time.monotonic() + 600
while (p := client.get_refresh_external_collection_progress(job_id=job_id)).state != "RefreshCompleted":
    if time.monotonic() > deadline:
        raise TimeoutError("External collection refresh timed out; inspect the job before retrying")
    if p.state == "RefreshFailed":
        raise RuntimeError(p.reason)
    time.sleep(1)

index_params = client.prepare_index_params()
index_params.add_index(field_name="emb", index_type="HNSW", metric_type="COSINE")
client.create_index(collection_name=name, index_params=index_params)
client.load_collection(collection_name=name)

hits = client.search(collection_name=name, data=[embed("rotate credentials")],  # your 1024-d embedding
                     anns_field="emb", limit=5, output_fields=["title"])
print(hits)

Format values: lance-table, iceberg-table, parquet, vortex. snapshot_id applies only to iceberg-table. extfs also accepts AWS IAM, Milvus global credentials, or an IAM role ARN; set endpoint_url for S3-compatible storage. Code requires a running server and your own embedding model; it is an example, not a benchmark.

Explore the concepts

Frequently asked questions

Does Apache Iceberg support vector search?

Iceberg has no vector type or index of its own, but Milvus 3.0+ serves one: create an external collection with "format": "iceberg-table", point it at the table's metadata JSON file and a snapshot_id, then refresh, index, and search. Embeddings stored as float lists map to FLOAT_VECTOR.

Can Milvus search a Lance dataset without importing it?

Yes. Point external_source at the Lance dataset directory, set "format": "lance-table", map fields with external_field, then refresh, index, and search. The rows stay in Lance, read-only; Milvus stores only the mapping, manifests, indexes, and caches.

Does Lance have its own vector index, and why index it with Milvus?

Lance datasets can carry native indexes (IVF_PQ, IVF_RQ, IVF_HNSW_SQ, IVF_HNSW_FLAT) that LanceDB queries in-process. Milvus builds its own HNSW, IVF or DiskANN index in Milvus storage, so one cluster serves many clients with filters, BM25 and hybrid search.

LanceDB vs Milvus: which should I use?

Both can keep data in Lance. LanceDB OSS is an embedded library that queries datasets in-process. Milvus runs as a distributed service that indexes Lance, Iceberg, Parquet, and Vortex sources and serves search to many clients.

How do you keep a vector index in sync with lake tables?

Call refresh_external_collection after the lake changes. It is asynchronous and incremental: unchanged data keeps its segments and indexes, and queries serve the previous state until the refresh completes.

Lance vs Iceberg vs Parquet: which should hold embeddings?

Milvus reads all three. Lance suits AI datasets needing fast random access, versioning and frequent column additions; Iceberg adds table-level snapshots, schema evolution and SQL-engine governance over Parquet files; Parquet alone has the broadest tool compatibility.

Does Zilliz Cloud support external collections?

Zilliz Cloud external collections are in Public Preview and supported only in on-demand compute databases. See the linked Zilliz Cloud documentation for current limits.

Sources and technical documentation (21)