Milvus vs LanceDB: format, embedded library, or distributed database?
Milvus vs LanceDB is really a comparison of layers: Lance is an open columnar format, LanceDB is the embedded database built over it, and Milvus is a distributed vector database that can serve Lance datasets in place or store data itself. LanceDB OSS runs inside one application process; Milvus runs as a service with storage-compute separation, replicas and multi-writer ingest. Keep Lance as your source of truth either way; choose LanceDB when one process owns the data, and Milvus when many services, writers or tenants share it.
- Apache-2.0, 40K+ GitHub stars
- Searches Lance datasets in place with external collections
- Milvus Lite: pip install, one local file
- Distributed: tens of billions of vectors
- Salesforce
- Walmart
- NVIDIA
- eBay
- IBM
- Cisco
- Roblox
- Exa.ai
- OpenEvidence
- Read.ai
- Plaud
- Filevine
Lance vs LanceDB: what is the difference?
Lance is an open, Arrow-native lakehouse format for multimodal AI (Apache-2.0, lance-format/lance). It is three specs in one: a file format whose column pages are laid out for fast random access, a table format that manages fragments, manifests, deletions, schema evolution and ACID commits as immutable versions, and index formats that make vector, full-text and scalar indexes first-class, versioned table objects.
LanceDB is the database built on top of Lance. LanceDB OSS is an embedded retrieval library with Python, TypeScript and Rust SDKs that organizes Lance datasets into named tables and runs search inside your process; LanceDB Cloud and Enterprise are the company's hosted and BYOC services that add always-on query nodes, plan executors and indexers over the same tables. Lance format vs Parquet is a storage question; LanceDB vs Milvus is a serving question.
| Lance (format) | LanceDB (database) | |
|---|---|---|
| What it is | Open columnar file, table and index formats on local disk or object storage | Embedded library (OSS) plus Cloud and Enterprise services built over Lance tables |
| What it holds | Fragments of column data, immutable manifests per version, deletion files, blobs for media | Named tables over Lance datasets, a catalog of namespaces, search and query APIs |
| Native indexes | Vector (IVF and HNSW layouts), full-text, BTree, Bitmap, Label List, Bloom, N-gram, JSON | Builds and queries those indexes; hybrid search with rerankers in the host process |
| Who reads it | pylance, DuckDB, Spark, Ray, Trino, Polars, PyTorch loaders, Milvus external collections | Your application through the LanceDB SDK, or Enterprise query nodes |
| Project | Apache-2.0, github.com/lance-format/lance | Apache-2.0 OSS, github.com/lancedb/lancedb; services from LanceDB Inc. |
Can Milvus use Lance as a storage format too?
Yes. A Milvus external collection (Milvus 3.0+) points external_source at a Lance dataset directory with format lance-table, maps each field to a source column with external_field, and then indexes and serves the data in place. There is no import and no second copy: the rows stay in Lance, read-only, and Milvus stores only the schema mapping, manifests, indexes and caches.
Your Lance datasets stay the source of truth; Milvus adds the serving layer. Spark, Ray or pylance keep writing fragments, and the same files remain readable by LanceDB, DuckDB and training jobs.
- Enable Storage V3 (common.storage.useLoonFFI: true), create the schema with external_source and external_spec, then refresh, index, load and search.
- Milvus builds its own HNSW, IVF or DiskANN index in Milvus storage and never writes into the dataset, so Lance's native indexes and versions are untouched.
- Refresh is explicit, asynchronous and incremental: new fragments are picked up, unchanged data keeps its segments and indexes, and queries serve the previous state until it completes.
- External fields can feed Functions, so BM25 sparse vectors or text embeddings are generated in Milvus without writing a column back to the table.
- The same mechanism reads Apache Iceberg tables (format iceberg-table with a snapshot_id), Parquet and Vortex; see vector search on the data lake.
Milvus and LanceDB: from embedded retrieval to distributed serving
For the embedded case, Milvus Lite is a pip install (pymilvus) that runs in-process and keeps data in a local file: MilvusClient("./milvus_demo.db"). It targets notebooks, laptops, edge devices and prototypes, and the same MilvusClient API moves unchanged to Standalone, Distributed and Zilliz Cloud. That is the LanceDB OSS shape: zero services, one process, local storage.
The core difference is what happens past one process. Milvus is a distributed system with storage-compute separation: stateless proxies, streaming nodes for writes and the WAL, query nodes for search, and data nodes for compaction and index build, all over shared object storage. Multiple clients write at once, replicas raise throughput and absorb failures, a query fans out across query nodes, and tiered storage keeps hot data in memory and local disk while the rest stays in object storage. LanceDB OSS is bound to its host process; its docs put OSS at millions of vectors on a single node and recommend LanceDB Enterprise, which adds its own query nodes, plan executors, indexers and a control plane, for hundreds of millions and beyond.
| Milvus (Lite to Distributed) | LanceDB OSS | LanceDB Enterprise | |
|---|---|---|---|
| Runs as | Lite: in-process; Distributed: stateless proxy, streaming, query and data nodes | Library inside your process | Query nodes, plan executors, indexers, control plane |
| Where data lives | Lite: local file; Distributed: object storage, WAL included (Woodpecker) | Lance tables on local NVMe, EBS, EFS or S3 / GCS / Azure | Lance tables and index artifacts in object storage |
| Concurrent writers | Multiple clients through proxies and the streaming WAL | Each add() commits a version; commit retries are bounded | Writes validated and committed by query nodes |
| Replicas and query fan-out | In-memory replicas for QPS and failover; one query spans many query nodes | The host process | Search traffic spread across multiple nodes |
| Hot and cold data | Tiered storage: lazy load, partial load, LRU eviction across memory, disk, object storage | Host memory and disk | Distributed NVMe cache |
| Documented scale | Lite: a few million; Standalone: 100 million; Distributed: tens of billions | Millions of vectors on a single node | Hundreds of millions to billions of rows |
| Deployment | pip, Docker, Helm or Milvus Operator; Zilliz Cloud managed | pip / npm / cargo | Managed or BYOC |
Documented scales are deployment guidance, not hard product limits; measure with your index, hardware and workload.
LanceDB vs Milvus features: what each does that the other does not
LanceDB's edge is the data layer: because the table is Lance, the same files flow through Arrow, pandas, Polars, DuckDB and PyTorch, every write is a time-travelable version, and images or audio sit next to their embeddings as blobs. Milvus's edge is the serving layer: server-side hybrid search with BM25, fused by RRF, weighted, decay or model rankers and composed with Function Chain; partition-key multi-tenancy for millions of tenants; RBAC; a broad index catalog including GPU, DiskANN and RaBitQ; streaming ingest that is searchable on write under a chosen consistency level; and external collections over Lance and Iceberg.
| Capability | LanceDB | Milvus |
|---|---|---|
| Arrow, pandas, Polars, DuckDB interop | Tables are Arrow-native Lance; read and write with pandas, PyArrow, Polars, DuckDB, dlt | Bulk import from Parquet, JSON, CSV; lake tables served via external collections |
| Zero-service local mode | OSS library in Python, TypeScript or Rust; nothing to run | Milvus Lite: pip install, local file, FLAT index, same API as the cluster |
| Dataset versioning and time travel | Every write is a Lance version; time travel, tags and branches | Snapshots and milvus-backup; lake versions stay with the lake |
| Multimodal blobs in the table | Images, video, audio and embeddings in one Lance table with a blob API | Vectors with scalar, JSON, array, text and StructArray fields; media stays in your object store or lake |
| Hybrid search and reranking | Vector plus FTS with a reranker in the host process (RRF, Cohere, cross-encoder, ColBERT) | Built-in BM25 and multi-vector fusion on the server: RRF, weighted, decay and model rankers, composable with Function Chain (Milvus 3.0+) |
| Multi-tenancy and access control | Namespaces organize many tables | Database, collection, partition or partition key for millions of tenants; RBAC with users, roles and privileges; shared collections require a tenant filter on every request |
| Index breadth | IVF_PQ, IVF_RQ, IVF_HNSW_SQ, IVF_HNSW_PQ, IVF_HNSW_FLAT; FTS; BTREE, BITMAP, LABEL_LIST | HNSW, HNSW_SQ/PQ/PRQ, IVF_FLAT/SQ8/PQ/RABITQ, DISKANN, SCANN, GPU_CAGRA, GPU_IVF; sparse, binary, MinHash; INVERTED, BITMAP, JSON path |
| Streaming ingest and consistency | Writes commit versions; readers refresh per read_consistency_interval; OSS calls optimize() to index new rows | Writes land in the streaming WAL and become searchable under the selected consistency level; Strong, Bounded (default), Session or Eventually per request |
| Lake formats served | Lance tables | External collections over Lance, Apache Iceberg, Parquet and Vortex, plus native collections |
When is LanceDB the better choice?
What is LanceDB good at? Removing a tier. For a single process that owns its data, LanceDB OSS gives you search, full-text and versioned storage from a library and a directory. If that process later needs to share the data with other services, the Lance files are already the kind of thing Milvus serves.
- A single-process app, notebook, CLI or edge function that wants search with no server.
- Local-first development where the dataset fits one machine and ships with the code.
- Lance-native pipelines: PyTorch, Ray or Spark jobs that train on and query the same versioned table.
- Multimodal datasets where images, audio and embeddings belong in one table with time travel.
- Teams that want Arrow, DuckDB and Polars on the same files as their vector search.
Which AI companies and enterprises run Milvus in production?
Milvus and Zilliz Cloud (managed Milvus) run in AI-native companies and large enterprises alike. The milvus.io adopter wall includes Salesforce, Walmart, NVIDIA, eBay, IBM, Cisco, Roblox, and Reddit, alongside Exa.ai, OpenEvidence, Read.ai, Notta.ai, Plaud, and Filevine. Published results:
- OpenEvidence: under 10 ms search latency and 4,000 QPS at peak on Zilliz Cloud.
- Plaud: 10B+ vector chunks per region with under 200 ms average recall latency.
- Exa: neural search latency cut from seconds to under 200 ms with hybrid search.
- Airtable: self-hosts Milvus on Kubernetes with the Milvus Operator for semantic search.
Customer stories
Filevine
30 billion vectors managed across millions of legal documents on Zilliz Cloud.
Read the storyMeituan
10B+ vectors in production on self-run Milvus, with 2B+ in a single collection.
Read the storyRead AI
20–50 ms retrieval latency on Milvus for millions of monthly users and million-scale active tenants.
Read the storyExplore the concepts
- How can Milvus search data stored in Lance?
- How do Lance, Iceberg, and Parquet play different roles in a vector lake?
- When should you use a Milvus external collection?
- What determines vector-database scalability?
- How do sharding and replication differ in a vector database?
- How do you choose a vector index for your workload?
- What is hybrid search, and when should you use it?
- How should you design multi-tenancy for vector search?
Frequently asked questions
What is the difference between Lance and LanceDB?
Lance is the open columnar format: files, versioned tables of fragments, and native vector, full-text and scalar indexes, all Arrow-native. LanceDB is the database built on it: an embedded OSS library that queries Lance tables in-process, plus LanceDB Cloud and Enterprise services. Milvus reads Lance too.
Can Milvus read Lance files?
Yes. A Milvus 3.0+ external collection points external_source at the Lance dataset directory with format lance-table, maps columns with external_field, then refreshes, indexes and serves it. Rows stay in Lance, read-only, with no second copy; Storage V3 (common.storage.useLoonFFI) must be enabled.
Is Milvus Lite like LanceDB?
In shape, yes: Milvus Lite is pip-installed, runs in your Python process and stores data in a local file, like LanceDB OSS. It is meant for prototypes and edge use with a FLAT index; the same MilvusClient code then runs on Milvus Standalone, Distributed or Zilliz Cloud.
Is LanceDB production-ready, and how far does it scale?
LanceDB's docs say OSS comfortably handles millions of vectors on a single node, with bounded commit retries under many concurrent writers, and recommend LanceDB Enterprise for hundreds of millions and beyond. Milvus Distributed is documented for tens of billions of vectors with replicas and multi-writer ingest.
Can LanceDB and Milvus share the same Lance data?
Yes. Milvus never writes into the dataset: it keeps its own indexes in Milvus storage, so LanceDB, DuckDB, Spark and training jobs keep reading and writing the same Lance files while Milvus serves online search over them.
What is the difference between Milvus and Zilliz Cloud?
Zilliz Cloud is fully managed Milvus from the team that created it, with the same client APIs, so code written against Milvus Lite or Standalone moves there without a rewrite. It also offers external collections over Lance, Iceberg, Parquet and Vortex.External collections are currently in Public Preview and supported only in on-demand compute databases.
Sources and technical documentation (58)
- https://milvus.io
- https://github.com/milvus-io/milvus
- https://milvus.io/docs/architecture_overview.md
- https://milvus.io/docs/install-overview.md
- https://milvus.io/docs/milvus_lite.md
- https://milvus.io/docs/replica.md
- https://milvus.io/docs/consistency.md
- https://milvus.io/docs/tiered-storage-overview.md
- https://milvus.io/docs/index-explained.md
- https://milvus.io/docs/full-text-search.md
- https://milvus.io/docs/multi-vector-search.md
- https://milvus.io/docs/reranking.md
- https://milvus.io/docs/decay-ranker-overview.md
- https://milvus.io/docs/model-ranker-overview.md
- https://milvus.io/docs/release_notes.md
- https://milvus.io/api-reference/restful/v3.0.x/v2/Vector%20(v2)/Search.md
- https://milvus.io/docs/multi_tenancy.md
- https://milvus.io/docs/rbac.md
- https://milvus.io/docs/json-indexing.md
- https://milvus.io/docs/woodpecker_architecture.md
- https://milvus.io/docs/create-an-external-collection.md
- https://milvus.io/docs/storage-v3.md
- https://milvus.io/docs/snapshots.md
- https://milvus.io/docs/prepare-source-data.md
- https://milvus.io/docs/array-of-structs.md
- https://github.com/zilliztech/milvus-backup
- https://lance.org/format/
- https://lance.org/format/index/
- https://github.com/lance-format/lance
- https://docs.lancedb.com/lance
- https://docs.lancedb.com/
- https://github.com/lancedb/lancedb
- https://docs.lancedb.com/faq/faq-oss
- https://docs.lancedb.com/enterprise
- https://docs.lancedb.com/enterprise/architecture
- https://docs.lancedb.com/storage
- https://docs.lancedb.com/tables/consistency
- https://docs.lancedb.com/tables/versioning
- https://docs.lancedb.com/tables/multimodal
- https://docs.lancedb.com/indexing/vector-index
- https://docs.lancedb.com/indexing/fts-index
- https://docs.lancedb.com/indexing/scalar-index
- https://docs.lancedb.com/indexing/reindexing
- https://docs.lancedb.com/search/hybrid-search
- https://docs.lancedb.com/namespaces
- https://docs.lancedb.com/integrations/data/duckdb
- https://docs.lancedb.com/integrations/data/polars_arrow
- https://docs.lancedb.com/integrations/data/pandas_and_pyarrow
- https://docs.lancedb.com/training/torch
- https://docs.zilliz.com/docs/external-collection-limits
- https://zilliz.com/customers/filevine
- https://zilliz.com/customers/openevidence
- https://zilliz.com/customers/exa
- https://zilliz.com/customers/airtable
- https://zilliz.com/customers/plaud
- https://zilliz.com/customers/read-ai
- https://zilliz.com/customers/meituan
- https://milvus.io/docs/quickstart.md