We Extracted OpenClaw’s Memory System and Open-Sourced It (memsearch)



If you've been building AI agents, you've probably hit this wall: your demo works great, but getting it into production is a whole different story.

We've covered agent memory management and reranking in earlier posts. Now let's tackle the bigger challenge—building agents that actually hold up in production.

Here's the reality: production environments are messy. A single agent rarely cuts it, which is why multi-agent systems are everywhere. But the frameworks available today tend to fall into two camps: lightweight ones that demo well but break under real load, or powerful ones that take forever to learn and build with.

I've been experimenting with [Agno](https://github.com/agno-agi/agno) recently, and it seems to strike a reasonable middle ground—focused on production readiness without excessive complexity. The project has gained over 37,000 GitHub stars in a few months, suggesting other developers find it useful as well.

In this post, I'll share what I learned while building a multi-agent system using Agno with [Milvus](https://milvus.io/) as the memory layer. We'll look at how Agno compares to alternatives such as LangGraph and walk through a complete implementation you can try yourself.

## What Is Agno?

[Agno](https://github.com/agno-agi/agno) is a multi-agent framework built specifically for production use. It has two distinct layers:

-   **Agno framework layer**: Where you define your agent logic
    
-   **AgentOS runtime layer**: Turns that logic into HTTP services you can actually deploy
    

Think of it this way: the framework layer defines *what* your agents should do, while AgentOS handles *how* that work gets executed and served.

### The Framework Layer

This is what you work with directly. It introduces three core concepts:

-   **Agent**: Handles a specific type of task
    
-   **Team**: Coordinates multiple agents to solve complex problems
    
-   **Workflow**: Defines execution order and structure
    

One thing I appreciated: you don't need to learn a new DSL or draw flowcharts. Agent behavior is defined using standard Python function calls. The framework handles LLM invocation, tool execution, and memory management.

### The AgentOS Runtime Layer

AgentOS is designed for high request volumes through async execution, and its stateless architecture makes scaling straightforward.

Key features include:

-   Built-in FastAPI integration for exposing agents as HTTP endpoints
    
-   Session management and streaming responses
    
-   Monitoring endpoints
    
-   Horizontal scaling support
    

In practice, AgentOS handles most of the infrastructure work, which lets you focus on the agent logic itself.

A high-level view of Agno’s architecture is shown below.

![](https://assets.zilliz.com/1_dfbf444ee6.png)

## Agno vs. LangGraph

To understand where Agno fits, let's compare it with LangGraph—one of the most widely used multi-agent frameworks.

[**LangGraph**](https://www.langchain.com/langgraph) uses a graph-based state machine. You model your entire agent workflow as a graph: steps are nodes, execution paths are edges. This works well when your process is fixed and strictly ordered. But for open-ended or conversational scenarios, it can feel restrictive. As interactions get more dynamic, maintaining a clean graph gets harder.

**Agno** takes a different approach. Instead of being a pure orchestration layer, it's an end-to-end system. Define your agent behavior, and AgentOS automatically exposes it as a production-ready HTTP service—with monitoring, scalability, and multi-turn conversation support built in. No separate API gateway, no custom session management, no extra operational tooling.

Here's a quick comparison:

| Dimension | LangGraph | Agno |
| --- | --- | --- |
| Orchestration model | Explicit graph definition using nodes and edges | Declarative workflows defined in Python |
| State management | Custom state classes defined and managed by developers | Built-in memory system |
| Debugging & observability | LangSmith (paid) | AgentOS UI (open source) |
| Runtime model | Integrated into an existing runtime | Standalone FastAPI-based service |
| Deployment complexity | Requires additional setup via LangServe | Works out of the box |

LangGraph gives you more flexibility and fine-grained control. Agno optimizes for faster time-to-production. The right choice depends on your project stage, existing infrastructure, and the level of customization you need. If you're unsure, running a small proof of concept with both is probably the most reliable way to decide.

## Choosing Milvus for the Agent Memory Layer

Once you've chosen a framework, the next decision is how to store memory and knowledge. We use Milvus for this. [Milvus](https://milvus.io/) is the most popular open-source vector database built for AI workloads with more than [42,000+ GitHub](https://github.com/milvus-io/milvus) stars. 

**Agno has native Milvus support.** The `agno.vectordb.milvus` module wraps production features like connection management, automatic retries, batch writes, and embedding generation. You don't need to build connection pools or handle network failures yourself—a few lines of Python give you a working vector memory layer.

**Milvus scales with your needs.** It supports three [deployment modes:](https://milvus.io/docs/install-overview.md)

-   **Milvus Lite**: Lightweight, file-based—great for local development and testing
    
-   **Standalone**: Single-server deployment for production workloads
    
-   **Distributed**: Full cluster for high-scale scenarios
    

You can start with Milvus Lite to validate your agent memory locally, then move to standalone or distributed as traffic grows—without changing your application code. This flexibility is especially useful when you're iterating quickly in early stages but need a clear path to scale later.

![](https://assets.zilliz.com/1_1_1en_e0294d0ffa.png)

## Step-by-Step: Building a Production-Ready Agno Agent with Milvus

Let's build a production-ready agent from scratch.

We'll start with a simple single-agent example to show the full workflow. Then we'll expand it into a multi-agent system. AgentOS will automatically package everything as a callable HTTP service.

### 1. Deploying Milvus Standalone with Docker

**(1) Download the Deployment Files**

```
**wget** **
<https://github.com/Milvus-io/Milvus/releases/download/v2.****5****.****12****/Milvus-standalone-docker-compose.yml> -O docker-compose.yml**
```

**(2) Start the Milvus Service**

```
docker-compose up -d
```

```
docker-compose ps -a
```

![](https://assets.zilliz.com/_80575354d3.png)

### 2. Core Implementation

```
import os
from pathlib import Path
from agno.os import AgentOS
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.knowledge.knowledge import Knowledge
from agno.vectordb.milvus import Milvus
from agno.knowledge.embedder.openai import OpenAIEmbedder
from agno.db.sqlite import SqliteDb
os.environ\["OPENAI_API_KEY"\] = "you-key-here"
data_dir = Path("./data")
data_dir.mkdir(exist_ok=True)
knowledge_base = Knowledge(
    contents_db=SqliteDb(
        db_file=str(data_dir / "knowledge_contents.db"),
        knowledge_table="knowledge_contents",
    ),
    vector_db=Milvus(
        collection="agno_knowledge",
        uri="http://192.168.x.x:19530",
        embedder=OpenAIEmbedder(id="text-embedding-3-small"),
    ),
)
*# Create Agent*
agent = Agent(
    name="Knowledge Assistant",
    model=OpenAIChat(id="gpt-4o"),
    instructions=\[
        "You are a knowledge base assistant that helps users query and manage knowledge base content.",
        "Answer questions in English.",
        "Always search the knowledge base before answering questions.",
        "If the knowledge base is empty, kindly prompt the user to upload documents."
    \],
    knowledge=knowledge_base,
    search_knowledge=True,
    db=SqliteDb(
        db_file=str(data_dir / "agent.db"),
        session_table="agent_sessions",
    ),
    add_history_to_context=True,
    markdown=True,
)
agent_os = AgentOS(agents=\[agent\])
app = agent_os.get_app()
if __name__ == "__main__":
    print("\n🚀 Starting service...")
    print("📍 http://localhost:7777")
    print("💡 Please upload documents to the knowledge base in the UI\n")
    agent_os.serve(app="knowledge_agent:app", port=7777, reload=False)
```

**(1) Running the Agent**

```
**python** **knowledge_agent.py**
```

![](https://assets.zilliz.com/1_df885706cf.png)

### 3. Connecting to the AgentOS Console

https://os.agno.com/

**(1) Create an Account and Sign In**

![](https://assets.zilliz.com/2_db0af51e58.png)

**(2) Connect Your Agent to AgentOS**

![](https://assets.zilliz.com/3_0a8c6f9436.png)

**(3) Configure the Exposed Port and Agent Name**

![](https://assets.zilliz.com/4_3844011799.png)

**(4) Add Documents and Index Them in Milvus**

![](https://assets.zilliz.com/5_776ea7ca11.png)

![](https://assets.zilliz.com/6_90b97c4660.png)

![](https://assets.zilliz.com/7_a98262d8c5.png)

![](https://assets.zilliz.com/8_58b7d77eea.png)

**(5) Test the Agent End to End**

![](https://assets.zilliz.com/_6e61038ba5.png)

In this setup, Milvus handles high-performance semantic retrieval. When the knowledge-base assistant receives a technical question, it invokes the `search_knowledge` tool to embed the query, retrieves the most relevant document chunks from Milvus, and uses those results as the basis for its response.

Milvus offers three deployment options, allowing you to choose an architecture that fits your operational requirements while keeping the application-level APIs consistent across all deployment modes.

The demo above shows the core retrieval and generation flow. To move this design into a production environment, however, several architectural aspects need to be discussed in more detail.

## How Retrieval Results Are Shared Across Agents

Agno's Team mode has a `share_member_interactions=True` option that allows later agents to inherit the full interaction history of earlier agents. In practice, this means that when the first agent retrieves information from Milvus, subsequent agents can reuse those results instead of running the same search again.

-   **The upside:** Retrieval costs are amortized across the team. One vector search supports multiple agents, reducing redundant queries.
    
-   **The downside:** Retrieval quality gets amplified. If the initial search returns incomplete or inaccurate results, that error propagates to every agent that depends on it.
    

This is why retrieval accuracy matters even more in multi-agent systems. A bad retrieval doesn't just degrade one agent's response—it affects the entire team.

  

Here's an example Team setup:

```
from agno.team import Team
analyst = Agent(
    name="Data Analyst",
    model=OpenAIChat(id="gpt-4o"),
    knowledge=knowledge_base,
    search_knowledge=True,
    instructions=\["Analyze data and extract key metrics"\]
)
writer = Agent(
    name="Report Writer",
    model=OpenAIChat(id="gpt-4o"),
    knowledge=knowledge_base,
    search_knowledge=True,
    instructions=\["Write reports based on the analysis results"\]
)
team = Team(
    agents=\[analyst, writer\],
    share_member_interactions=True,  *# Share knowledge retrieval results*
)
```

## Why Agno and Milvus Are Layered Separately

In this architecture, **Agno** sits at the conversation and orchestration layer. It is responsible for managing dialogue flow, coordinating agents, and maintaining conversational state, with session history persisted in a relational database. The system’s actual domain knowledge—such as product documentation and technical reports—is handled separately and stored as vector embeddings in **Milvus**. This clear division keeps conversational logic and knowledge storage fully decoupled.

Why this matters operationally:

-   **Independent scaling**: As Agno demand grows, add more Agno instances. As query volume grows, expand Milvus by adding query nodes. Each layer scales in isolation.
    
-   **Different hardware needs**: Agno is CPU- and memory-bound (LLM inference, workflow execution). Milvus is optimized for high-throughput vector retrieval (disk I/O, sometimes GPU acceleration). Separating them prevents resource contention.
    
-   **Cost optimization**: You can tune and allocate resources for each layer independently.
    

This layered approach gives you a more efficient, resilient, and production-ready architecture.

## What to Monitor When Using Agno with Milvus

Agno has built-in evaluation capabilities, but adding Milvus expands what you should watch. Based on our experience, focus on three areas:

-   **Retrieval quality**: Are the documents Milvus returns actually relevant to the query, or just superficially similar at the vector level?
    
-   **Answer faithfulness**: Is the final response grounded in the retrieved content, or is the LLM generating unsupported claims?
    
-   **End-to-end latency breakdown**: Don't just track total response time. Break it down by stage—embedding generation, vector search, context assembly, LLM inference—so you can identify where slowdowns occur.
    

**A practical example:** When your Milvus collection grows from 1 million to 10 million vectors, you might notice retrieval latency creeping up. That's usually a signal to tune index parameters (like `nlist` and `nprobe`) or consider moving from standalone to a distributed deployment.

## Conclusion

Building production-ready agent systems takes more than wiring together LLM calls and retrieval demos. You need clear architectural boundaries, infrastructure that scales independently, and observability to catch issues early.

In this post, I walked through how Agno and Milvus can work together: Agno for multi-agent orchestration, Milvus for scalable memory and semantic retrieval. By keeping these layers separate, you can move from prototype to production without rewriting core logic—and scale each component as needed.

If you're experimenting with similar setups, I'd be curious to hear what's working for you.

**Questions about Milvus?** Join our [Slack channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session.

How to Build Production-Ready Multi-Agent Systems with Agno and Milvus


What Is OpenClaw? Complete Guide to the Open-Source AI Agent


OpenClaw (Formerly Clawdbot & Moltbot) Explained: A Complete Guide to the Autonomous AI Agent



If you've been on tech Twitter, Hacker News, or Discord this week, you've seen it. A lobster emoji 🦞, screenshots of tasks being completed, and one bold claim: an AI that doesn't just *talk*—it actually *does*.

![](https://assets.zilliz.com/ST_1_567975a33f.png)

It got weirder over the weekend. Entrepreneur Matt Schlicht launched [Moltbook](https://moltbook.com)—a Reddit-style social network where only AI agents can post, and humans can only watch. Within days, over 1.5 million agents signed up. They formed communities, debated philosophy, complained about their human operators, and even founded their own religion called "Crustafarianism." Yes, really.
![](https://assets.zilliz.com/ST_2_b570b3e59b.png)

Welcome to the OpenClaw craze.

The hype is so real that Cloudflare's stock jumped 14% simply because developers use its infrastructure to run applications. Mac Mini sales reportedly spiked as people buy dedicated hardware for their new AI employee. And the GitHub repo? Over [150,000 stars](https://github.com/openclaw/openclaw) in just a few weeks.

So naturally, we had to show you how to set up your own OpenClaw instance—and connect it to Slack so you can boss around your AI assistant from your favorite messaging app.

## What Is OpenClaw?

[OpenClaw](https://openclaw.ai/) (formerly known as Clawdbot/Moltbot) is an open-source, autonomous AI agent that runs locally on user machines and performs real-world tasks via messaging apps such as WhatsApp, Telegram, and Discord. It automates digital workflows—such as managing emails, browsing the web, or scheduling meetings—by connecting to LLMs like Claude or ChatGPT. 

In short, it's like having a 24/7 digital assistant that can think, respond, and actually get stuff done.


## Setting Up OpenClaw as a Slack-Based AI Assistant

Imagine having a bot in your Slack workspace that can instantly answer questions about your product, help debug user issues, or point teammates to the right documentation—without anyone having to stop what they're doing. For us, that could mean faster support for the Milvus community: a bot that answers common questions ("How do I create a collection?"), helps troubleshoot errors, or summarizes release notes on demand. For your team, it might be onboarding new engineers, handling internal FAQs, or automating repetitive DevOps tasks. The use cases are wide open.

In this tutorial, we'll walk through the basics: installing OpenClaw on your machine and connecting it to Slack. Once that's done, you'll have a working AI assistant ready to customize for whatever you need.

### Prerequisites

-   A Mac or Linux machine
    
-   An [Anthropic API key](https://console.anthropic.com/) (or Claude Code CLI access)
    
-   A Slack workspace where you can install apps
    

That's it. Let's get started.

### Step 1: Install OpenClaw

Run the installer:

curl -fsSL https://molt.bot/install.sh | bash
![](https://assets.zilliz.com/ST_3_fc80684811.png)

  
When prompted, select **Yes** to continue.
![](https://assets.zilliz.com/ST_4_8004e87516.png)

  
Then, choose **QuickStart** mode.
![](https://assets.zilliz.com/ST_5_b5803c1d89.png)

  
### Step 2: Choose Your LLM

The installer will ask you to pick a model provider. We're using Anthropic with the Claude Code CLI for authentication.

1.  Select **Anthropic** as the provider
![](https://assets.zilliz.com/ST_6_a593124f6c.png)


2.  Complete the verification in your browser when prompted.
![](https://assets.zilliz.com/ST_7_410c1a39d3.png)


3.  Choose **anthropic/claude-opus-4-5-20251101** as your default model
![](https://assets.zilliz.com/ST_8_0c22bf5a16.png)

  
### Step 3: Set Up Slack

When asked to select a channel, choose **Slack.**
![](https://assets.zilliz.com/ST_9_cd4dfa5053.png)

  
Proceed to name your bot. We called ours "Clawdbot_Milvus."
![](https://assets.zilliz.com/ST_10_89c79ccd0d.png)

  
Now you'll need to create a Slack app and grab two tokens. Here's how:
![](https://assets.zilliz.com/ST_11_50df3aec5d.png)

**3.1 Create a Slack App**

Go to the [Slack API website](https://api.slack.com/apps?new_app=1) and create a new app from scratch.
![](https://assets.zilliz.com/ST_12_21987505d5.png)

  
Give it a name and select the workspace you want to use.
![](https://assets.zilliz.com/ST_13_7fce24b5c7.png)

**3.2 Set Bot Permissions**

In the sidebar, click **OAuth & Permissions**. Scroll down to **Bot Token Scopes** and add the permissions your bot needs.
![](https://assets.zilliz.com/ST_14_b08d66b55a.png)


**3.3 Enable Socket Mode**

Click **Socket Mode** in the sidebar and toggle it on.
![](https://assets.zilliz.com/ST_15_11545f95f8.png)


This will generate an **App-Level Token** (starts with `xapp-`). Copy it somewhere safe.
![](https://assets.zilliz.com/ST_16_c446eefd7d.png)


**3.4 Enable Event Subscriptions**

Go to **Event Subscriptions** and toggle it on.
![](https://assets.zilliz.com/ST_17_98387d6226.png)


Then choose which events your bot should subscribe to.
![](https://assets.zilliz.com/ST_18_b2a16d7786.png)

  
**3.5 Install the App**

Click **Install App** in the sidebar, then **Request to Install** (or install directly if you're a workspace admin).
![](https://assets.zilliz.com/ST_19_a5e76d0d33.png)

  
Once approved, you'll see your **Bot User OAuth Token** (starts with `xoxb-`). Copy this as well.
![](https://assets.zilliz.com/oauth_tokens_2e75e66f89.png)


### Step 4: Configure OpenClaw

Back in the OpenClaw CLI:

1.  Enter your **Bot User OAuth Token** (`xoxb-...`)
    
2.  Enter your **App-Level Token** (`xapp-...`)
![](https://assets.zilliz.com/ST_21_bd1629fb6a.png)


3.  Select which Slack channels the bot can access
![](https://assets.zilliz.com/ST_22_a1b682fa84.png)


4.  Skip skills configuration for now—you can always add them later
![](https://assets.zilliz.com/ST_23_cc4855ecfd.png)


5.  Select **Restart** to apply your changes
    

### Step 5: Try It Out

Head over to Slack and send your bot a message. If everything's set up correctly, OpenClaw will respond and be ready to run tasks on your machine.
![](https://assets.zilliz.com/ST_24_2864a88ce9.png)


### Tips

1.  Run `clawdbot dashboard` to manage settings through a web interface
![](https://assets.zilliz.com/ST_25_67b337b1d9.png)


2.  If something goes wrong, check the logs for error details
![](https://assets.zilliz.com/ST_26_a62b5669ee.png)


## A Word of Caution

OpenClaw is powerful—and that's exactly why you should be careful. "Actually does things" means it can execute real commands on your machine. That's the whole point, but it comes with risk.

**The good news:**

-   It's open source, so the code is auditable
    
-   It runs locally, so your data isn't on someone else's server
    
-   You control what permissions it has
    

**The not-so-good news:**

-   Prompt injection is a real risk—a malicious message could potentially trick the bot into running unintended commands
    
-   Scammers have already created fake OpenClaw repos and tokens, so be careful what you download
    

**Our advice:**

-   Don't run this on your primary machine. Use a VM, a spare laptop, or a dedicated server.
    
-   Don't grant more permissions than you need.
    
-   Don't use this in production yet. It's new. Treat it like the experiment it is.
    
-   Stick to official sources: [@openclaw](https://x.com/openclaw) on X and [OpenClaw](https://github.com/openclaw). 
    

Once you give an LLM the ability to execute commands, there's no such thing as 100% secure. That's not an OpenClaw problem—that's the nature of agentic AI. Just be smart about it.

## What's Next?

Congratulations! You now have a local AI assistant running on your own infrastructure, accessible through Slack. Your data stays yours, and you've got a tireless helper ready to automate the repetitive stuff.

From here, you can:

-   Install more [Skills](https://docs.molt.bot/skills) to expand what OpenClaw can do
    
-   Set up scheduled tasks so it works proactively
    
-   Connect other messaging platforms like Telegram or Discord
    
-   Explore the [Milvus](https://milvus.io/) ecosystem for AI search capabilities
    

**Have questions or want to share what you're building?**

-   Join the [Milvus Slack community](https://milvus.io/slack) to connect with other developers
    
-   Book our [Milvus Office Hours](https://milvus.io/office-hours) for live Q&A with the team
    

Happy hacking! 🦞

OpenClaw Tutorial: Connect to Slack for Local AI Assistant


Step-by-Step Guide to Setting Up OpenClaw (Previously Clawdbot/Moltbot) with Slack


Build Clawdbot-Style AI Agents with LangGraph & Milvus


Why Clawdbot Went Viral — And How to Build Production-Ready Long-Running Agents with LangGraph and Milvus


How Claude Code Manages Local Storage for AI Agents


Why Claude Code Feels So Stable: A Developer’s Deep Dive into Its Local Storage Design



[Claude Cowork](https://support.claude.com/en/articles/13345190-getting-started-with-cowork) is a new agent feature in the Claude Desktop app. From a developer’s point of view, it’s basically an automated task runner wrapped around the model: it can read, modify, and generate local files, and it can plan multi-step tasks without you having to manually prompt for each step. Think of it as the same loop behind Claude Code, but exposed to the desktop instead of the terminal. 

Cowork’s key capability is its ability to run for extended periods without losing state. It doesn’t hit the usual conversation timeout or context reset. It can keep working, track intermediate results, and reuse previous information across sessions. That gives the impression of “long-term memory,” even though the underlying mechanics are more like persistent task state + contextual carryover. Either way, the experience is different from the traditional chat model, where everything resets unless you build your own memory layer.

This brings up two practical questions for developers:

1.  **If the model can already remember past information, where does RAG or agentic RAG still fit in? Will RAG be replaced?** 
    
2.  **If we want a local, Cowork-style agent, how do we implement long-term memory ourselves?**
    

The rest of this article addresses these questions in detail and explains how vector databases fit into this new “model memory” landscape.

## Claude Cowork vs. RAG: What’s the Difference? 

As I mentioned previously, Claude Cowork is an agent mode inside Claude Desktop that can read and write local files, break tasks into smaller steps, and keep working without losing state. It maintains its own working context, so multi-hour tasks don’t reset like a normal chat session.

**RAG** (Retrieval-Augmented Generation) solves a different problem: giving a model access to external knowledge. You index your data into a vector database, retrieve relevant chunks for each query, and feed them into the model. It’s widely used because it provides LLM applications with a form of “long-term memory” for documents, logs, product data, and more.

If both systems help a model “remember,” what’s the actual difference?

### How Cowork Handles Memory

Cowork’s memory is read-write. The agent decides which information from the current task or conversation is relevant, stores it as memory entries, and retrieves it later as the task progresses. This allows Cowork to maintain continuity across long-running workflows — especially ones that produce new intermediate state as they progress.

### How RAG and Agentic RAG Handle Memory

Standard RAG is query-driven retrieval: the user asks something, the system fetches relevant documents, and the model uses them to answer. The retrieval corpus stays stable and versioned, and developers control exactly what enters it.

Modern agentic RAG extends this pattern. The model can decide when to retrieve information, what to retrieve, and how to use it during the planning or execution of a workflow. These systems can run long tasks and call tools, similar to Cowork. But even with agentic RAG, the retrieval layer remains knowledge-oriented rather than state-oriented. The agent retrieves authoritative facts; it doesn’t write its evolving task state back into the corpus.

Another way to look at it:

-   **Cowork’s memory is task-driven:** the agent writes and reads its own evolving state.
    
-   **RAG is knowledge-driven:** the system retrieves established information that the model should rely on.
    

## Reverse-Engineering Claude Cowork: How It Builds Long-Running Agent Memory

Cowork gets a lot of hype because it handles multi-step tasks without constantly forgetting what it was doing. From a developer's perspective, I am wondering **how it keeps state across such long sessions?** Anthropic hasn’t published the internals, but based on earlier dev experiments with Claude’s memory module, we can piece together a decent mental model.

Claude seems to rely on a hybrid setup: **a persistent long-term memory layer plus on-demand retrieval tools.** Instead of stuffing the full conversation into every request, Claude selectively pulls in past context only when it decides it’s relevant. This lets the model keep accuracy high without blowing through tokens every turn.

If you break down the request structure, it roughly looks like this:

```
[0] Static system instructions
[1] User memory (long-term)
[2] Retrieved / pruned conversation history
[3] Current user message
```

The interesting behavior isn’t the structure itself — it’s how the model decides what to update and when to run retrieval.

### User Memory: The Persistent Layer

Claude keeps a long-term memory store that updates over time. And unlike ChatGPT’s more predictable memory system, Claude’s feels a bit more “alive.” It stores memories in XML-ish blocks and updates them in two ways: 

-   **Implicit updates:** Sometimes the model just decides something is a stable preference or fact and quietly writes it to memory. These updates aren’t instantaneous; they show up after a few turns, and older memories can fade out if the related conversation disappears.
    
-   **Explicit updates:** Users can directly modify memory with the `memory_user_edits` tool (“remember X,” “forget Y”). These writes are immediate and behave more like a CRUD operation.
    

Claude is running background heuristics to decide what’s worth persisting, and it’s not waiting for explicit instructions.

### Conversation Retrieval: The On-Demand Part

Claude does *not* keep a rolling summary like many LLM systems. Instead, it has a toolbox of retrieval functions it can call whenever it thinks it’s missing context. These retrieval calls don’t happen every turn — the model triggers them based on its own internal judgment. 

The standout is `conversation_search`. When the user says something vague like “that project from last month,” Claude often fires this tool to dig up relevant turns. What’s notable is that it still works when the phrasing is ambiguous or in a different language. That pretty clearly implies:

-   Some kind of semantic matching (embeddings)
    
-   Probably combined with normalization or lightweight translation
    
-   Keyword search layered in for precision
    

Basically, this looks a lot like a miniature RAG system bundled inside the model’s toolset.

### How Claude’s Retrieval Behavior Differs From Basic History Buffers

From testing and logs, a few patterns stand out:

-   **Retrieval isn’t automatic.** The model chooses when to call it. If it thinks it already has enough context, it won’t even bother.
    
-   **Retrieved chunks include** *both* **user and assistant messages.** That’s useful — it keeps more nuance than user-only summaries.
    
-   **Token usage stays sane.** Because history isn’t injected every turn, long sessions don’t balloon unpredictably.
    

Overall, it feels like a retrieval-augmented LLM, except the retrieval happens as part of the model’s own reasoning loop.

This architecture is clever, but not free:

-   Retrieval adds latency and more “moving parts” (indexing, ranking, re-ranking).
    
-   The model occasionally misjudges whether it needs context, which means you see the classic “LLM forgetfulness” even though the data *was* available.
    
-   Debugging becomes trickier because model behavior depends on invisible tool triggers, not just prompt input.
    

### Claude Cowork vs Claude Codex in handling long-term memory 

In contrast to Claude’s retrieval-heavy setup, ChatGPT handles memory in a much more structured and predictable way. Instead of doing semantic lookups or treating old conversations like a mini vector store, ChatGPT injects memory directly into each session through the following layered components: 

-   User memory
    
-   Session metadata
    
-   Current session messages
    

**User Memory**

User Memory is the main long-term storage layer—the part that persists across sessions and can be edited by the user. It stores pretty standard things: name, background, ongoing projects, learning preferences, that kind of stuff. Every new conversation gets this block injected at the start, so the model always starts with a consistent view of the user.

ChatGPT updates this layer in two ways:

-   **Explicit updates:** Users can tell the model to “remember this” or “forget that,” and the memory changes immediately. This is basically a CRUD API that the model exposes through natural language.
    
-   **Implicit updates:** If the model spots information that fits OpenAI’s rules for long-term memory—like a job title or a preference—and the user hasn’t disabled memory, it will quietly add it on its own.
    

From a developer angle, this layer is simple, deterministic, and easy to reason about. No embedding lookups, no heuristics about what to fetch.

**Session Metadata**

Session Metadata sits at the opposite end of the spectrum. It’s short-lived, non-persistent, and only injected once at the start of a session. Think of it as environment variables for the conversation. This includes things like:

-   what device you’re on
    
-   account/subscription state
    
-   rough usage patterns (active days, model distribution, average conversation length)
    

This metadata helps the model shape responses for the current environment—e.g., writing shorter answers on mobile—without polluting long-term memory.

**Current Session Messages**

This is the standard sliding-window history: all messages in the current conversation until the token limit is reached. When the window gets too large, older turns drop off automatically.

Crucially, this eviction **does not** touch User Memory or cross-session summaries. Only the local conversation history shrinks.

The biggest divergence from Claude appears in how ChatGPT handles “recent but not current” conversations. Claude will call a search tool to retrieve past context if it thinks it’s relevant. ChatGPT does not do that.

Instead, ChatGPT keeps a very lightweight **cross-session summary** that gets injected into every conversation. A few key details about this layer:

-   It summarizes **only user messages**, not assistant messages.
    
-   It stores a very small set of items—roughly 15—just enough to capture stable themes or interests.
    
-   It does **no embedding computation, no similarity ranking, and no retrieval calls**. It’s basically pre-chewed context, not dynamic lookup.
    

From an engineering perspective, this approach trades flexibility for predictability. There’s no chance of a weird retrieval failure, and inference latency stays stable because nothing is being fetched on the fly. The downside is that ChatGPT won’t pull in some random message from six months ago unless it made it into the summary layer.

## Challenges to Making Agent Memory Writable

When an agent moves from **read-only memory** (typical RAG) to **writable memory**—where it can log user actions, decisions, and preferences—the complexity jumps quickly. You’re no longer just retrieving documents; you’re maintaining a growing state on which the model depends. 

A writable memory system has to solve three real problems:

1.  **What to remember:** The agent needs rules for deciding which events, preferences, or observations are worth keeping. Without this, memory either explodes in size or fills with noise.
    
2.  **How to store and tier memory:** Not all memory is equal. Recent items, long-term facts, and ephemeral notes all need different storage layers, retention policies, and indexing strategies.
    
3.  **How to write fast without breaking retrieval:** Memory must be written continuously, but frequent updates can degrade index quality or slow queries if the system isn’t designed for high-throughput inserts.
    

### Challenge 1: What Is Worth Remembering?

Not everything a user does should end up in long-term memory. If someone creates a temp file and deletes it five minutes later, recording that forever doesn’t help anyone. This is the core difficulty: **how does the system decide what actually matters?** 

**(1) Common ways to judge importance**

Teams usually rely on a mix of heuristics:

-   **Time-based**: recent actions matter more than old ones
    
-   **Frequency-based**: files or actions accessed repeatedly are more important
    
-   **Type-based**: some objects are inherently more important (for example, project config files vs. cache files)
    

**(2) When rules conflict**

These signals often conflict. A file created last week but edited heavily today—should age or activity win? There’s no single “correct” answer, which is why importance scoring tends to get messy fast.

**(3) How vector databases help** 

Vector databases give you mechanisms to enforce importance rules without manual cleanup:

-   **TTL:** Milvus can automatically remove data after a set time
    
-   **Decay:** older vectors can be down-weighted so they naturally fade from retrieval
    

### Challenge 2: Memory Tiering in Practice

As agents run longer, memory piles up. Keeping everything in fast storage isn’t sustainable, so the system needs a way to split memory into **hot** (frequently accessed) and **cold** (rarely accessed) tiers.

**(1) Deciding When Memory Becomes Cold**

In this model, *hot memory* refers to data kept in RAM for low-latency access, while *cold memory* refers to data moved to disk or object storage to reduce cost. 

Deciding when memory becomes cold can be handled in different ways. Some systems use lightweight models to estimate the semantic importance of an action or file based on its meaning and recent usage. Others rely on simple, rule-based logic, such as moving memory that has not been accessed for 30 days or has not appeared in retrieval results for a week. Users may also explicitly mark certain files or actions as important, ensuring they always remain hot.

**(2) Where Hot and Cold Memory Are Stored**

Once classified, hot and cold memories are stored differently. Hot memory stays in RAM and is used for frequently accessed content, such as active task context or recent user actions. Cold memory is moved to disk or object storage systems like S3, where access is slower but storage costs are much lower. This trade-off works well because cold memory is rarely needed and is typically accessed only for long-term reference.

**(3) How Vector Databases Help**

**Milvus and Zilliz Cloud** support this pattern by enabling hot–cold tiered storage while maintaining a single query interface, so frequently accessed vectors stay in memory and older data moves to lower-cost storage automatically.

### Challenge 3: How Fast Should Memory Be Written?

Traditional RAG systems usually write data in batches. Indexes are rebuilt offline—often overnight—and only become searchable later. This approach works for static knowledge bases, but it does not fit agent memory.

**(1) Why Agent Memory Needs Real-Time Writes**

Agent memory must capture user actions as they happen. If an action is not recorded immediately, the next conversation turn may lack critical context. For this reason, writable memory systems require real-time writes rather than delayed, offline updates.

**(2) The Tension Between Write Speed and Retrieval Quality**

Real-time memory demands very low write latency. At the same time, high-quality retrieval depends on well-built indexes, and index construction takes time. Rebuilding an index for every write is too expensive, but delaying indexing means newly written data remains temporarily invisible to retrieval. This trade-off sits at the center of writable memory design.

**(3) How Vector Databases Help**

Vector databases address this problem by decoupling writing from indexing. A common solution is to stream writes and perform incremental index builds. Using **Milvus** as an example, new data is first written to an in-memory buffer, allowing the system to handle high-frequency writes efficiently. Even before a full index is built, buffered data can be queried within seconds through dynamic merging or approximate search.

When the buffer reaches a predefined threshold, the system builds indexes in batches and persists them. This improves long-term retrieval performance without blocking real-time writes. By separating fast ingestion from slower index construction, Milvus achieves a practical balance between write speed and search quality that works well for agent memory.

## Conclusion 

Cowork gives us a glimpse of a new class of agents—persistent, stateful, and able to carry context across long timelines. But it also makes something else clear: long-term memory is only half of the picture. To build production-ready agents that are both autonomous and reliable, we still need structured retrieval over large, evolving knowledge bases.

RAG handles the world’s facts; writable memory handles the agent’s internal state. And vector databases sit at the intersection, providing indexing, hybrid search, and scalable storage that enable both layers to work together.

As long-running agents continue to mature, their architectures will likely converge on this hybrid design. Cowork is a strong signal of where things are heading—not toward a world without RAG, but toward agents with richer memory stacks powered by vector databases underneath.

If you want to explore these ideas or get help with your own setup, **join our** [Slack Channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) to chat with Milvus engineers. And for more hands-on guidance, you can always **book a** [**Milvus Office Hours**](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) **session.**


RAG vs Long-Running Agents: Is RAG Obsolete? 


Is RAG Becoming Outdated Now That Long-Running Agents Like Claude Cowork Are Emerging?



Tool usage is a big part of making an agent work. The agent needs to choose the right tool, decide when to call it, and format the inputs correctly. On paper that sounds straightforward, but once you start building real systems, you find a lot of edge cases and failure modes.

Many teams use MCP-style tool definitions to organize this, but MCP has some rough edges. The model has to reason over all tools at once, and there isn’t much structure to guide its decisions. On top of that, every tool definition has to live in the context window. Some of these are large — the GitHub MCP is around 26k tokens — which eats context before the agent even starts doing real work.

Anthropic introduced [**Skills**](https://github.com/anthropics/skills?tab=readme-ov-file) to improve this situation. Skills are smaller, more focused, and easier to load on demand. Instead of dumping everything into context, you package domain logic, workflows, or scripts into compact units that the agent can pull in only when needed.

In this post, I’ll go over how Anthropic Skills work and then walk through building a simple Skill in Claude Code that turns natural language into a [Milvus](https://milvus.io/)-backed knowledge base — a quick setup for RAG without extra wiring.

## What Are Anthropic Skills?

[Anthropic Skills](https://milvus.io/blog/is-mcp-already-outdated-the-real-reason-anthropic-shipped-skills-and-how-to-pair-them-with-milvus.md) (or Agent Skills) are just folders that bundle the instructions, scripts, and reference files an agent needs to handle a specific task. Think of them as small, self-contained capability packs. A Skill might define how to generate a report, run an analysis, or follow a particular workflow or set of rules.

The key idea is that Skills are modular and can be loaded on demand. Instead of stuffing huge tool definitions into the context window, the agent pulls in only the Skill it needs. This keeps context usage low while giving the model clear guidance on what tools exist, when to call them, and how to execute each step.

The format is intentionally simple, and because of that, it’s already supported or easily adapted across a bunch of developer tools — Claude Code, Cursor, VS Code extensions, GitHub integrations, Codex-style setups, and so on.

A Skill follows a consistent folder structure:

```
skill-name/

├── SKILL.md       # Required: Skill instructions and metadata

├── scripts/         # Optional: helper scripts

├── templates/       # Optional: document templates

└── resources/       # Optional: reference materials
```

**1.** `SKILL.md` **(Core File)**

This is the execution guide for the agent—the document that tells the agent exactly how the task should be carried out. It defines the Skill’s metadata (such as name, description, and trigger keywords), the execution flow, and default settings. In this file, you should clearly describe:

- **When the Skill should run:** For example, trigger the Skill when the user input includes a phrase like “process CSV files with Python.”
    
- **How the task should be performed:** Lay out the execution steps in order, such as: interpret the user’s request → call preprocessing scripts from the `scripts/` directory → generate the required code → format the output using templates from `templates/`.
    
- **Rules and constraints:** Specify details such as coding conventions, output formats, and how errors should be handled.
    

**2.** `scripts/` **(Execution Scripts)**

This directory contains prewritten scripts in languages such as Python, Shell, or Node.js. The agent can call these scripts directly, instead of generating the same code repeatedly at runtime. Typical examples include `create_collection.py` and `check_env.py`.

**3.** `templates/` **(Document Templates)**

Reusable template files that the agent can use to generate customized content. Common examples include report templates or configuration templates.

**4.** `resources/` **(Reference Materials)**

Reference documents the agent can consult during execution, such as API documentation, technical specifications, or best-practice guides.

Overall, this structure mirrors how work is handed off to a new teammate: `SKILL.md` explains the job, `scripts/` provide ready-to-use tools, `templates/` define standard formats, and `resources/` supply background information. With all of this in place, the agent can execute the task reliably and with minimal guesswork.

## Hands-on Tutorial: Creating a Custom Skill for a Milvus-Powered RAG System

In this section, we’ll walk through building a custom Skill that can set up a Milvus collection and assemble a full RAG pipeline from plain natural-language instructions. The goal is to skip all the usual setup work — no manual schema design, no index configuration, no boilerplate code. You tell the agent what you want, and the Skill handles the Milvus pieces for you.

### Design Overview

![](https://assets.zilliz.com/design_overview_d4c886291b.PNG)

### Prerequisites

  

| Component | Requirement |
| --- | --- |
| CLI | `claude-code` |
| Models | GLM 4.7, OpenAI |
| Container | Docker |
| Milvus | 2.6.8 |
| Model Configuration Platform | CC-Switch |
| Package Manager | npm |
| Development Language | Python |

  

### Step 1: Environment Setup

**Install** `claude-code`

```
npm install -g @anthropic-ai/claude-code
```

**Install CC-Switch**

**Note:** CC-Switch is a model-switching tool that makes it easy to switch between different model APIs when running AI models locally.

Project repository: [https://github.com/farion1231/cc-switch](https://github.com/farion1231/cc-switch)

**Select Claude and Add an API Key**

![](https://assets.zilliz.com/1_0cdfab2e54.PNG)

![](https://assets.zilliz.com/2_615ee13649.PNG)

**Check the Current Status**

![](https://assets.zilliz.com/3_f1c13da1fe.PNG)

**Deploy and Start Milvus-Standalone**

```
# Download docker-compose.yml

wget https://github.com/milvus-io/milvus/releases/download/v2.6.8/milvus-standalone-docker-compose.yml -O docker-compose.yml

  

# Start Milvus (check port mapping: 19530:19530)

docker-compose up -d

  

# Verify that the services are running

docker ps | grep milvus

# You should see three containers: milvus-standalone, milvus-etcd, milvus-minio
```

![](https://assets.zilliz.com/code1_9c6a1a7f93.PNG)

**Configure the OpenAI API Key**

```
# Add this to ~/.bashrc or ~/.zshrc

OPENAI_API_KEY=your_openai_api_key_here
```

### Step 2: Create the Custom Skill for Milvus

**Create the Directory Structure**

```
cd ~/.claude/skills/

mkdir -p milvus-skills/example milvus-skills/scripts
```

**Initialize** `SKILL.md`

**Note:** SKILL.md serves as the agent’s execution guide. It defines what the Skill does and how it should be triggered.

```
name: milvus-collection-builder

description: Create Milvus collections using natural language, supporting both RAG and text search scenarios
```

**Write the Core Scripts**

| Script Type | File Name | Purpose |
| --- | --- | --- |
| Environment check | `check_env.py` | Checks the Python version, required dependencies, and the Milvus connection |
| Intent parsing | `intent_parser.py` | Converts requests like “build a RAG database” into a structured intent such as `scene=rag` |
| Collection creation | `milvus_builder.py` | The core builder that generates the collection schema and index configuration |
| Data ingestion | `insert_milvus_data.py` | Loads documents, chunks them, generates embeddings, and writes data into Milvus |
| Example 1 | `basic_text_search.py` | Demonstrates how to create a document search system |
| Example 2 | `rag_knowledge_base.py` | Demonstrates how to build a complete RAG knowledge base |

These scripts show how to turn a Milvus-focused Skill into something practical: a working document search system and an intelligent Q&A (RAG) setup.

### Step 3: Enable the Skill and Run a Test

**Describe the Request in Natural Language**

```
"I want to build an RAG system."
```

![](https://assets.zilliz.com/test1_64fd549573.PNG)

**RAG System Created**

![](https://assets.zilliz.com/test2_80656d59b1.png)

**Insert Sample Data**

![](https://assets.zilliz.com/test3_392753eb73.png)

**Run a Query**

![](https://assets.zilliz.com/test4_75e23c6a3a.png)

## Conclusion

In this tutorial, we walked through building a Milvus-powered RAG system using a custom Skill. The goal wasn’t just to show another way to call Milvus—it was to show how Skills can turn what is normally a multi-step, configuration-heavy setup into something you can reuse and iterate on. Instead of manually defining schemas, tuning indexes, or stitching together workflow code, the Skill handles most of the boilerplate so you can focus on the parts of RAG that actually matter.

This is only the start. A full RAG pipeline has plenty of moving pieces: preprocessing, chunking, hybrid search settings, reranking, evaluation, and more. All of these can be packaged as separate Skills and composed depending on your use case. If your team has internal standards for vector dimensions, index params, prompt templates, or retrieval logic, Skills are a clean way to encode that knowledge and make it repeatable.

For new developers, this lowers the entry barrier—no need to learn every detail of Milvus before getting something running. For experienced teams, it cuts down on repeated setup and helps keep projects consistent across environments. Skills won’t replace thoughtful system design, but they remove a lot of unnecessary friction.

👉 The full implementation is available in the [open-source repository](https://github.com/yinmin2020/open-milvus-skills), and you can explore more community-built examples in the [Skill marketplace](https://skillsmp.com/).

## Stay tuned!

We’re also working on introducing official Milvus and Zilliz Cloud Skills that cover common RAG patterns and production best practices. If you have ideas or specific workflows you want supported, join our [Slack Channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) and chat with our engineers. And if you want guidance for your own setup, you can always book a [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session.

Create a Custom Anthropic Skill for Milvus to Quickly Spin Up RAG


How Anthropic Skills Change Agent Tooling — and How to Build a Custom Skill for Milvus to Quickly Spin Up RAG



As AI development accelerates, more teams are discovering that building an AI assistant doesn’t necessarily require a software engineering background. The people who need assistants the most—product teams, operations, support, researchers—often know exactly what the agent should do, but not how to implement it in code. Traditional “no-code” tools tried to bridge that gap with drag-and-drop canvases, yet they collapse the moment you need real agent behavior: multi-step reasoning, tool use, or persistent memory. 

The newly released [**LangSmith Agent Builder**](https://www.langchain.com/langsmith/agent-builder) takes a different approach. Instead of designing workflows, you describe the agent’s goals and available tools in plain language, and the runtime handles the decision-making. No flowcharts, no scripting—just clear intent.

But intent alone doesn’t produce an intelligent assistant. **Memory** does. This is where [**Milvus**](https://milvus.io/), the widely adopted open-source vector database, provides the foundation. By storing documents and conversation history as embeddings, Milvus allows your agent to recall context, retrieve relevant information, and respond accurately at scale. 

This guide walks through how to build a production-ready, memory-enabled AI assistant using **LangSmith Agent Builder + Milvus**, all without writing a single line of code.

## What is LangSmith Agent Builder and How It Works? 

Just as its name reveals, [LangSmith Agent Builder](https://www.google.com/search?q=LangSmith+Agent+Builder&oq=what+is+LangSmith+Agent+Builder&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQABiABDIHCAIQABiABDIHCAMQABiABDIHCAQQABiABDIHCAUQABiABDIHCAYQABiABDIHCAcQABiABDIGCAgQABgDMggICRAAGBYYHtIBCTI1OTJqMGoxNagCCLACAfEF2Mylr_IuXLk&sourceid=chrome&ie=UTF-8&ved=2ahUKEwjV1LfvxZ6SAxVFsFYBHYzTJAsQgK4QegQIARAB) is a no-code tool from LangChain that lets you build, deploy, and manage AI agents using plain language. Instead of writing logic or designing visual flows, you explain what the agent should do, what tools it can use, and how it should behave. The system then handles the hard parts—generating prompts, selecting tools, wiring components together, and enabling memory. 

![](https://assets.zilliz.com/_57c5cee35b.png)

Unlike traditional no-code or workflow tools, Agent Builder doesn’t have drag-and-drop canvas and no node library. You interact with it the same way you would with ChatGPT. Describe what you want to build, answer a few clarifying questions, and the Builder produces a fully functioning agent based on your intent.  

Behind the scenes, that agent is constructed from four core building blocks.

![](https://assets.zilliz.com/640_05b90b1f3d.webp)

-   **Prompt:** The prompt is the agent’s brain, defining its goals, constraints, and decision logic. LangSmith Agent Builder uses meta-prompting to build this automatically: you describe what you want, it asks clarifying questions, and your answers are synthesized into a detailed, production-ready system prompt. Instead of hand-writing logic, you simply express intent.
-   **Tools:** Tools let the agent take action—sending emails, posting to Slack, creating calendar events, searching data, or calling APIs. Agent Builder integrates these tools through the Model Context Protocol (MCP), which provides a secure, extensible way to expose capabilities. Users can rely on built-in integrations or add custom MCP servers, including Milvus [MCP server](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)s for vector search and long-term memory.
-   **Triggers:** Triggers define when an agent runs. In addition to manual execution, you can attach agents to schedules or external events so they automatically respond to messages, emails, or webhook activity. When a trigger fires, Agent Builder starts a new execution thread and runs the agent’s logic, enabling continuous, event-driven behavior.
-   **Subagents:** Subagents break complex tasks into smaller, specialized units. A primary agent can delegate work to subagents—each with its own prompt and toolset—so tasks like data retrieval, summarization, or formatting are handled by dedicated helpers. This avoids a single overloaded prompt and creates a more modular, scalable agent architecture.

## How Does an Agent Remember Your Preferences?

What makes Agent Builder unique is how it treats *memory*. Instead of stuffing preferences into chat history, the agent can update its own behavior rules while running. If you say, “From now on, end every Slack message with a poem,” the agent doesn’t treat that as a one-off request—it stores it as a persistent preference that applies in future runs.

Under the hood, the agent keeps an internal memory file—essentially its evolving system prompt. Each time it starts, it reads this file to decide how to behave. When you give corrections or constraints, the agent edits the file by adding structured rules like “Always close the briefing with a short uplifting poem.” This approach is far more stable than relying on conversation history because the agent actively rewrites its operating instructions rather than burying your preferences inside a transcript.

This design comes from DeepAgents’ FilesystemMiddleware but is fully abstracted in Agent Builder. You never touch files directly: you express updates in natural language, and the system handles the edits behind the scenes. If you need more control, you can plug in a custom MCP server or drop to the DeepAgents layer for advanced memory customization.

## Hands-on Demo: Building a Milvus Assistant in 10 Minutes using Agent Builder 

Now that we’ve covered the design philosophy behind Agent Builder, let’s walk through the full build process with a hands-on example. Our goal is to create an intelligent assistant that can answer Milvus-related technical questions, search the official documentation, and remember user preferences over time. 

### Step 1. Sign In to the LangChain Website

![](https://assets.zilliz.com/640_1_b3c461d39b.webp)

### Step 2. Set Up Your Anthropic API Key

**Note:** Anthropic is supported by default. You can also use a custom model, as long as its type is included in the list officially supported by LangChain.

![](https://assets.zilliz.com/640_2_c04400695e.webp)

**1. Add an API Key** 

![](https://assets.zilliz.com/640_3_11db4b3824.webp)

**2. Enter and Save the API Key**

![](https://assets.zilliz.com/640_4_abfc27d796.webp)

### Step 3. Create a New Agent

**Note:** Click **Learn More** to view the usage documentation.

![](https://assets.zilliz.com/640_5_e90bf254f2.webp)

![](https://assets.zilliz.com/640_6_7c839d96f3.webp)

1.  **Configure a Custom Model (Optional)**

![](https://assets.zilliz.com/640_7_0dfd5ff561.webp)

**(1) Enter Parameters and Save**

![](https://assets.zilliz.com/640_8_85f9e3008f.webp)

![](https://assets.zilliz.com/640_9_0d5d0c062c.webp)

### Step 4. Describe Your Requirements to Create the Agent

**Note:** Create the agent using a natural language description.

```

I need a Milvus technical consultant to help me answer technical questions about vector databases. 

Search the official documentation and remember my preference for the index type I use. 

```

![](https://assets.zilliz.com/640_10_0e033a5200.webp)

1.  **The System Asks Follow-Up Questions to Refine Requirements**

Question 1: Select the Milvus index types you want the agent to remember

![](https://assets.zilliz.com/640_11_050ac891f0.webp)
  
Question 2: Choose how the agent should handle technical questions
![](https://assets.zilliz.com/640_12_d1d6d4f2ed.webp)
  
Question 3: Specify whether the agent should focus on guidance for a specific Milvus version
![](https://assets.zilliz.com/640_13_5d60df75e9.webp)

### Step 5. Review and Confirm the Generated Agent

**Note:** The system automatically generates the agent configuration.

![](https://assets.zilliz.com/640_14_8a596ae853.webp)

Before creating the agent, you can review its metadata, tools, and prompts. Once everything looks correct, click **Create** to proceed.

![](https://assets.zilliz.com/640_15_5c0b27aca7.webp)

![](https://assets.zilliz.com/640_16_998921b071.webp)

### Step 6. Explore the Interface and Feature Areas

After the agent is created, you’ll see three functional areas in the lower-left corner of the interface:

**(1) Triggers**

Triggers define when the agent should run, either in response to external events or on a schedule:

-   **Slack:** Activate the agent when a message arrives in a specific channel
-   **Gmail:** Trigger the agent when a new email is received
-   **Cron:** Run the agent on a scheduled interval

![](https://assets.zilliz.com/640_17_b77c603413.webp)

**(2) Toolbox**

This is the set of tools the agent can call. In the example shown, the three tools are generated automatically during creation, and you can add more by clicking **Add tool**.
![](https://assets.zilliz.com/640_18_94637d4548.webp)
 
**If your agent needs vector search capabilities—such as semantic search across large volumes of technical documentation—you can deploy Milvus’s MCP Server** and add it here using the **MCP** button. Make sure the MCP server is running **at a reachable network endpoint**; otherwise, Agent Builder won’t be able to invoke it.

![](https://assets.zilliz.com/640_19_94fe99a3b8.webp)
![](https://assets.zilliz.com/640_20_f887a8fbda.webp)

**(3) Sub-agents**

Create independent agent modules dedicated to specific subtasks, enabling a modular system design.

### Step 7. Test the Agent

Click **Test** in the top-right corner to enter testing mode. Below is a sample of the test results.

![](https://assets.zilliz.com/640_22_527619519b.webp)

![](https://assets.zilliz.com/640_23_639d40c40d.webp)  

![](https://assets.zilliz.com/640_24_42a71d2592.webp)

![](https://assets.zilliz.com/640_25_8ab35e15f8.webp)

## Agent Builder vs. DeepAgents: Which One Should You Choose?

LangChain offers multiple agent frameworks, and the right choice depends on how much control you need. [DeepAgents](https://www.google.com/search?q=DeepAgents&newwindow=1&sca_esv=0e7ec9ce2aa7d5b4&sxsrf=ANbL-n5pe1KqjmJVjQCqmc3jneYhmGGOUg%3A1769066335766&ei=X89xab21Lp3a1e8Ppam06Ag&ved=2ahUKEwio15nYzZ6SAxU_mq8BHcf3BqUQgK4QegQIARAB&uact=5&oq=what+is+DeepAgents&gs_lp=Egxnd3Mtd2l6LXNlcnAiEndoYXQgaXMgRGVlcEFnZW50czIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzINEAAYgAQYsAMYQxiKBTINEAAYgAQYsAMYQxiKBUi8BlAYWL8FcAF4AZABAJgBqgKgAbAKqgEFMC4yLjS4AQPIAQD4AQGYAgGgAgyYAwCIBgGQBgqSBwExoAehHrIHALgHAMIHAzMtMcgHCYAIAA&sclient=gws-wiz-serp) is agent building tool. It is used to build autonomous, long-running AI agents that handle complex, multi-step tasks. Built on LangGraph, it supports advanced planning, file-based context management, and subagent orchestration—making it ideal for long-horizon or production-grade projects.

So how does that compare to **Agent Builder**, and when should you use each?

**Agent Builder** focuses on simplicity and speed. It abstracts away most implementation details, letting you describe your agent in natural language, configure tools, and run it immediately. Memory, tool use, and human-in-the-loop workflows are handled for you. This makes Agent Builder perfect for rapid prototyping, internal tools, and early-stage validation where ease of use matters more than granular control.

**DeepAgents**, by contrast, is designed for scenarios where you need full control over memory, execution, and infrastructure. You can customize middleware, integrate any Python tool, modify the storage backend (including persisting memory in [Milvus](https://milvus.io/blog)), and explicitly manage the agent’s state graph. The trade-off is engineering effort—you write code, manage dependencies, and handle failure modes yourself—but you get a fully customizable agent stack.

Importantly, **Agent Builder and DeepAgents are not separate ecosystems—they form a single continuum**. Agent Builder is built on top of DeepAgents. That means you can start with a quick prototype in Agent Builder, then drop into DeepAgents when you need more flexibility, without rewriting everything from scratch. The reverse also works: patterns built in DeepAgents can be packaged as Agent Builder templates so non-technical users can reuse them.

## Conclusion

Thanks to the development of AI, building AI agents no longer requires complex workflows or heavy engineering. With LangSmith Agent Builder, you can create stateful, long-running assistants using natural language alone. You focus on describing what the agent should do, while the system handles planning, tool execution, and ongoing memory updates.

Paired with [Milvus](https://milvus.io/blog), these agents gain reliable, persistent memory for semantic search, preference tracking, and long-term context across sessions. Whether you’re validating an idea or deploying a scalable system, LangSmith Agent Builder and Milvus provide a simple, flexible foundation for agents that don’t just respond—they remember and improve over time.

Have questions or want a deeper walkthrough? Join our [Slack channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session for personalized guidance.

Build AI Agents in 10 Minutes with LangSmith Agent Builder & Milvus


Building AI Agents in 10 Minutes Using Natural Language with LangSmith Agent Builder + Milvus



[ScaNN](https://github.com/google-research/google-research/tree/master/scann) is Google’s answer to a familiar challenge in large-scale vector search: **how to increase query throughput and reduce memory usage without taking an unacceptable hit to result quality.** Conceptually, ScaNN starts from the classic IVF+PQ recipe—coarse clustering plus aggressive product quantization—but layers on two important innovations that meaningfully shift the performance frontier:

- **A score-aware quantization objective** that better preserves the relative ordering of true neighbors, improving ranking quality even under heavy compression.

- **FastScan** is a SIMD-optimized 4-bit PQ lookup path that reduces the traditional memory-load bottleneck by keeping more work inside CPU registers.

In practice, it is a strong choice when you are okay with trading some recall for high QPS and a much smaller memory footprint (often compressing vectors to ~1/16 of the original size), such as in recall-insensitive recommendation workloads.

In this post, we’ll revisit IVFPQ as the baseline, explore the key optimizations ScaNN introduces on top of it, and wrap up with experimental results that ground the discussion in measured performance.

## IVFPQ Recap

ScaNN was proposed by Google in 2020, and the paper reports a 3× performance improvement over HNSW on the GloVe dataset. You can refer to the [original paper](https://arxiv.org/pdf/1908.10396.pdf) and the [open-source implementation](https://github.com/google-research/google-research/tree/master/scann) for details. 

Before introducing ScaNN, we’ll briefly recap IVFPQ, since ScaNN is built on top of the same overall framework.

**IVFPQ stands for Inverted File with Product Quantization**, an algorithm used for efficient and large-scale Approximate Nearest Neighbor (ANN) search in high-dimensional vector databases. It is a hybrid approach that combines two techniques, the **inverted file index (IVF)** and **product quantization (PQ)**, to balance search speed, memory usage, and accuracy. 

### How IVFPQ Works

The process involves two main steps during indexing and searching: 

- IVF layer: vectors are clustered into `nlist` inverted lists (clusters). At query time, you visit only a subset of clusters (`nprobe`) to trade off recall and latency/throughput.

![](https://assets.zilliz.com/ivf1_5e3d29c392.png)

- PQ layer: within each visited cluster, each D-dimensional vector is split into m subvectors, each of dimension (D/m). Each subvector is quantized by assigning it to the nearest centroid in its sub-codebook. If a sub-codebook has 256 centroids, each subvector can be represented by a `uint8` code (an ID in [0, 255]).

![](https://assets.zilliz.com/pq2_6695c9cc6f.png)

Distance computation can then be rewritten as the sum over subvectors:

**D(q, X) = D(q, u0) + D(q, u1) + D(q, u2) + ... + D(q, un)**

**= L(q, id1) + L(q, id2) + L(q, id3) + ... + L(q, idn)**

Here, L represents a lookup table. At query time, the lookup table is constructed, recording the distance between the query and each quantized subvector. All subsequent distance computations are converted into table lookups followed by summation.

![](https://assets.zilliz.com/query_vector3_75d47bdd53.png)

For example, for 128-dimensional vectors split into 32 subvectors of 4 dimensions each, if each subvector is encoded by a `uint8` ID, the storage cost per vector drops from (128 x 4) bytes to (32 x 1) bytes—a 1/16 reduction.

## ScaNN Optimizations Based on IVFPQ

In summary, ScaNN improves IVFPQ in two aspects:

1.  Quantization: ScaNN proposes an objective beyond simply replacing each subvector with its nearest k-means centroid (i.e., minimizing reconstruction error).
    
2.  Lookup efficiency: ScaNN accelerates LUT-based search, which is often memory-bound, via a SIMD-friendly FastScan path.
    

### Score-aware Quantization Loss

![](https://assets.zilliz.com/text_765e538864.png) 

![](https://assets.zilliz.com/score_aware_quantization_loss_d9d36223c2.png)

![](https://assets.zilliz.com/text2_dd95418ba9.png)

![](https://assets.zilliz.com/quantizer_f0e39762a4.png)

Since the analysis is based on the IP metric, after ScaNN decomposes the quantization error into parallel and perpendicular components, only the parallel component affects the result, so a larger penalty term should be applied. Consequently, the loss function can be rewritten as follows:

$$\begin{aligned} \ell\left(x_i, \tilde{x}_i, w\right) &=h_{\|}\left(w,\left\|x_i\right\|\right)\left\|r_{\|}\left(x_i, \tilde{x}_i\right)\right\|^2 +h_{\perp}\left(w,\left\|x_i\right\|\right)\left\|r_{\perp}\left(x_i, \tilde{x}_i\right)\right\|^2 \end{aligned}$$

The figure below shows a two-dimensional example illustrating that the error caused by the parallel component is larger and can lead to incorrect nearest neighbor results, thus warranting a more severe penalty.

![The left figure shows poor quantization because the parallel offset affects the final result, while the right figure shows better quantization.](https://assets.zilliz.com/quantization_cp_81e23dd1df.png)

### 4-bit PQ FastScan

Let's first review the PQ computation process: during querying, the distances between the query and subvector cluster centers are pre-computed to construct a lookup table. Distance computation is then performed through table lookups to obtain segment distances and sum them.

![](https://assets.zilliz.com/PQ_Fast_Scan1_248c53bc93.png)

However, frequent memory reads still become a performance bottleneck. If the lookup table can be made small enough to fit in registers, memory read operations can be transformed into efficient CPU SIMD instructions.

First, each subvector is clustered into 16 classes, so a 4-bit value can represent a cluster center—this is the origin of the name "4-bit PQ." Then, distances typically represented as floats are further converted to uint8 using Scalar Quantization (SQ). This way, the lookup table for one subvector can be stored in a register using 16 × 8 = 128 bits.

![](https://assets.zilliz.com/PQ_Fast_Scan2_07b9589195.png)

Finally, let's examine the register storage layout (using AVX2 instruction set as an example): the subvectors of 32 vectors are placed in a 128-bit register, combined with the lookup table. The "lookup" operation can then be efficiently completed using a single SIMD shuffle CPU instruction.

![register layout](https://assets.zilliz.com/register_layout_b2bf8c2b2b.png)

![SIMD Shuffle for Lookup](https://assets.zilliz.com/SIMD_Shuffle_for_Lookup_c86e6d9657.png)

Here's an interesting observation: the ScaNN paper focuses entirely on the first optimization, which is reasonable since it can be considered an algorithm paper emphasizing mathematical derivations. However, the experimental results presented in the paper are remarkably impressive.

![The experimental results presented in the ScaNN paper.](https://assets.zilliz.com/experimental_results_a46ec830a7.png)

Intuitively, optimizing the loss alone should not produce such dramatic effects. Another [blog](https://medium.com/@kumon/similarity-search-scann-and-4-bit-pq-fastscan-ab98766b32bd) has also pointed this out—what really makes the difference is the 4-bit PQ FastScan portion.

## Experimental Results

Using the vector database benchmark tool [VectorDBBench](https://github.com/zilliztech/VectorDBBench), we conducted a simple test. ScaNN's performance advantage over traditional IVFFLAT and IVF_PQ is quite evident. After integration into Milvus, on the Cohere1M dataset at the same recall rate, QPS can reach 5x that of IVFFLAT and 6x that of IVF_PQ.

However, QPS is slightly lower than that of graph-based indexes like HNSW, so it is not the first choice for high-QPS use cases. But for scenarios with lower recall, it is acceptable (such as in some recommendation systems), using ScaNN without loading raw data can achieve impressive QPS with an extremely low memory footprint (1/16 of the original data), making it an excellent index choice.

| Index_Type | Case | QPS | latency(p99) | recall | memory |
| --- | --- | --- | --- | --- | --- |
| IVFFLAT | Performance1M | 266 | 0.0173s | 0.9544 | 3G |
| HNSW | Performance1M | 1885 | 0.0054s | 0.9438 | 3.24G |
| IVF_PQ | Performance1M | 208 | 0.0292s | 0.928 | 0.375G |
| ScaNN（with_raw_data: true） | Performance1M | 1215 | 0.0069s | 0.9389 | 3.186G |
| ScaNN（with_raw_data: false） | Performance1M | 1265 | 0.0071s | 0.7066 | 0.186G |

## Conclusion

ScaNN builds on the familiar IVFPQ framework but pushes it significantly further through deep engineering work in both quantization and low-level lookup acceleration. By aligning the quantization objective with ranking quality and eliminating memory bottlenecks in the inner loop, ScaNN combines score-aware quantization with a 4-bit PQ FastScan path that turns a traditionally memory-bound process into an efficient, SIMD-friendly computation.

In practice, this gives ScaNN a clear and well-defined niche. It is not intended to replace graph-based indexes like HNSW in high-recall settings. Instead, for recall-insensitive workloads with tight memory budgets, ScaNN delivers high throughput with a very small footprint. In our experiments, after integration into Milvus VectorDB, ScaNN achieved roughly 5× the QPS of IVFFLAT on the Cohere1M dataset, making it a strong choice for high-throughput, low-latency ANN retrieval where compression and efficiency matter more than perfect recall.

If you’re interested in exploring ScaNN further or discussing index selection in real-world systems, join our [Slack Channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) to chat with our engineers and other AI engineers in the community. You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


A Brief Introduction to the ScaNN Index and How It Improves IVFPQ


A Brief Introduction to the ScaNN Index 




## The Problem: RAG Noise and Token Waste

**Vector search** is a solid foundation for RAG systems—enterprise assistants, AI agents, customer support bots, and more. It reliably finds the documents that matter. But retrieval alone doesn’t solve the context problem. Even well-tuned indexes return chunks that are broadly relevant, while only a small fraction of the sentences inside those chunks actually answer the query.

In production systems, this gap shows up immediately. A single query may pull in dozens of documents, each thousands of tokens long. Only a handful of sentences contain the actual signal; the rest is context that bloats token usage, slows inference, and often distracts the LLM. The problem becomes even more obvious in agent workflows, where the queries themselves are the output of multi-step reasoning and only match small parts of the retrieved text.

This creates a clear need for a model that can _**identify and highlight**_ _the useful sentences and ignore the rest_—essentially, sentence-level relevance filtering, or what many teams refer to as [**context pruning**](https://milvus.io/blog/llm-context-pruning-a-developers-guide-to-better-rag-and-agentic-ai-results.md). The goal is simple: keep the parts that matter and drop the noise before it ever reaches the LLM.

Traditional keyword-based highlighting can’t solve this problem. For example, if a user asks, “How do I improve Python code execution efficiency?”, a keyword highlighter will pick out “Python” and “efficiency,” but miss the sentence that actually answers the question—“Use NumPy vectorized operations instead of loops”—because it shares no keywords with the query. What we need instead is semantic understanding, not string matching.

## A Semantic Highlighting Model for RAG Noise Filtering and Context Pruning 

To make this easy for RAG builders, we trained and open-sourced a [**Semantic Highlighting model**](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1) that identifies and highlights the sentences in retrieved documents that are more semantically aligned with the query. The model currently delivers the state-of-the-art performance on both English and Chinese and is designed to slot directly into existing RAG pipelines.

![](https://assets.zilliz.com/context_pruning_80f7b16280.png)

**Model Details**

*   **HuggingFace:** [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1)
    
*   **License:** MIT (commercial-friendly)
    
*   **Architecture:** 0.6B encoder-only model based on BGE-M3 Reranker v2
    
*   **Context Window:** 8192 tokens
    
*   **Supported Languages:** English and Chinese
    

Semantic Highlighting provides the relevance signals needed to select only the useful parts of long retrieved documents. In practice, this model enables:

*   **Improved interpretability**, showing which parts of a document actually matter
    
*   **70–80% reduction in token cost** by sending only highlighted sentences to the LLM
    
*   **Better answer quality**, since the model sees less irrelevant context
    
*   **Easier debugging**, because engineers can inspect sentence-level matches directly
    

### Evaluation Results: Achieving SOTA Performance

We evaluated our Semantic Highlighting model across multiple datasets spanning both English and Chinese, in both in-domain and out-of-domain conditions. 

The benchmark suites include:

*   **English multi-span QA:** multispanqa
    
*   **English out-of-domain Wikipedia:** wikitext2
    
*   **Chinese multi-span QA:** multispanqa_zh
    
*   **Chinese out-of-domain Wikipedia:** wikitext2_zh
    

![](https://assets.zilliz.com/benchmarking_results_25545c952f.png)

Evaluated models include:

*   Open Provence series
    
*   Naver's Provence/XProvence series
    
*   OpenSearch's semantic-highlighter
    
*   Our trained bilingual model: [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1)
    

Across all four datasets, our model achieves the top ranking. More importantly, it is the _only_ model that performs consistently well on both English and Chinese. Competing models either focus exclusively on English or show clear performance drops on Chinese text.

## How We Built This Semantic Highlighting Model

Training a model for this task isn’t the hard part; training a _good_ model that handles the earlier problems and delivers near-SOTA performance is where the real work happens. Our approach focused on two things:

*   **Model architecture:** use an encoder-only design for fast inference.
    
*   **Training data:** generate high-quality relevance labels using reasoning-capable LLMs and scale data generation with local inference frameworks.
    

### Model Architecture

We built the model as a lightweight **encoder-only** network that treats context pruning as a **token-level relevance scoring task**. This design is inspired by [Provence](https://arxiv.org/html/2501.16214v1), a context-pruning approach introduced by Naver at ICLR 2025, which reframes pruning from “choose the right chunk” to “score every token.” That framing aligns naturally with semantic highlighting, where fine-grained signals are essential.

Encoder-only models aren’t the newest architecture, but they remain extremely practical here: they’re fast, easy to scale, and can produce relevance scores for all token positions in parallel. For a production RAG system, that speed advantage matters far more than using a larger decoder model.

Once we compute token-level relevance scores, we aggregate them into **sentence-level** scores. This step turns noisy token signals into a stable, interpretable relevance metric. Sentences above a configurable threshold are highlighted; everything else is filtered out. This produces a simple and reliable mechanism for selecting the sentences that actually matter to the query.

### Inference Process

At runtime, our semantic highlighting model follows a simple pipeline:

1.  **Input—** The process starts with a user query. Retrieved documents are treated as candidate context for relevance evaluation.
    
2.  **Model Processing—** The query and context are concatenated into a single sequence: [BOS] + Query + Context
    
3.  **Token Scoring—** Each token in the context is assigned a relevance score between 0 and 1, reflecting how strongly it is related to the query.
    
4.  **Sentence Aggregation—** Token scores are aggregated at the sentence level, typically by averaging, to produce a relevance score for each sentence.
    
5.  **Threshold Filtering—** Sentences with scores above a configurable threshold are highlighted and retained, while low-scoring sentences are filtered out before being passed to the downstream LLM.
    

![](https://assets.zilliz.com/semantic_highlighting_workflows_db3d12a666.png)

### Base Model: BGE-M3 Reranker v2

We selected BGE-M3 Reranker v2 as our base model for several reasons:

1.  It employs an Encoder architecture suitable for token and sentence scoring
    
2.  Supports multiple languages with optimization for both English and Chinese
    
3.  Provides an 8192-token context window appropriate for longer RAG documents
    
4.  Maintains 0.6B parameters—strong enough without being computationally heavy
    
5.  Ensures sufficient world knowledge in the base model
    
6.  Trained for reranking, which closely aligns with relevance judgment tasks
    

## Training Data: LLM Annotation with Reasoning

Once we finalized the model architecture, the next challenge was building a dataset that would actually train a reliable model. We started by looking at how Open Provence handles this. Their approach uses public QA datasets and a small LLM to label which sentences are relevant. It scales well and is easy to automate, which made it a good baseline for us.

But we quickly ran into the same issue they describe: if you ask an LLM to output sentence-level labels directly, the results aren’t always stable. Some labels are correct, others are questionable, and it’s hard to clean things up afterward. Fully manual annotation wasn’t an option either—we needed far more data than we could ever label by hand.

To improve stability without sacrificing scalability, we made one change: the LLM must provide a short reasoning snippet for every label it outputs. Each training example includes the query, the document, the sentence spans, and a brief explanation of why a sentence is relevant or irrelevant. This small adjustment made the annotations much more consistent and gave us something concrete to reference when validating or debugging the dataset.

Including the reasoning turned out to be surprisingly valuable:

*   **Higher annotation quality:** Writing out the reasoning works as a self-check, which reduces random or inconsistent labels.
    
*   **Better observability:** We can see _why_ a sentence was selected instead of treating the label as a black box.
    
*   **Easier debugging:** When something looks wrong, the reasoning makes it easy to spot whether the issue is the prompt, the domain, or the annotation logic.
    
*   **Reusable data:** Even if we switch to a different labeling model in the future, the reasoning traces remain useful for re-labeling or auditing.
    

The annotation workflow looks like this:

![](https://assets.zilliz.com/annotation_data_generation_ff93eb18f4.png)

### Qwen3 8B for Annotation

For annotation, we chose Qwen3 8B because it natively supports a “thinking mode” via outputs, making it much easier to extract consistent reasoning traces. Smaller models didn’t give us stable labels, and larger models were slower and unnecessarily expensive for this kind of pipeline. Qwen3 8B hit the right balance between quality, speed, and cost.

We ran all annotations using a **local vLLM service** instead of cloud APIs. This gave us high throughput, predictable performance, and much lower cost—essentially trading GPU time for API token fees, which is the better deal when generating millions of samples.

### Dataset Scale

In total, we built **over 5 million bilingual training samples**, split roughly evenly between English and Chinese.

*   **English sources:** MS MARCO, Natural Questions, GooAQ
    
*   **Chinese sources:** DuReader, Chinese Wikipedia, mmarco_chinese
    

Part of the dataset comes from re-annotating existing data used by projects like Open Provence. The rest was generated from raw corpora by first creating query–context pairs and then labeling them with our reasoning-based pipeline.

All annotated training data is also available on HuggingFace for community development and training reference: [Zilliz Datasets](https://huggingface.co/zilliz/datasets)

![](https://assets.zilliz.com/zilliz_datasets_dd91330d4d.png)

### Training Method

Once the model architecture and dataset were ready, we trained the model on **8× A100 GPUs** for three epochs, which took roughly **9 hours** end-to-end. 

**Note:** The training only targeted the **Pruning Head**, which is responsible for the semantic highlighting task. We did not train the **Rerank Head**, since focusing solely on the pruning objective yielded better results for sentence-level relevance scoring.

## Real-World Case Study

Benchmarks only tell part of the story, so here’s a real example that shows how the model behaves on a common edge case: when the retrieved text contains both the correct answer and a very tempting distractor.

**Query:** _Who wrote The Killing of a Sacred Deer?_

**Context (5 sentences):**


```
1\. The Killing of a Sacred Deer is a 2017 psychological horror film directed by Yorgos Lanthimos,

   with a screenplay by Lanthimos and Efthymis Filippou.

2\. The film stars Colin Farrell, Nicole Kidman, Barry Keoghan, Raffey Cassidy,

   Sunny Suljic, Alicia Silverstone, and Bill Camp.

3\. The story is based on the ancient Greek playwright Euripides' play Iphigenia in Aulis.

4\. The film tells the story of a cardiac surgeon (Farrell) who secretly

   befriends a teenager (Keoghan) connected to his past.

5\. He introduces the boy to his family, who then mysteriously fall ill.
```

Correct answer: Sentence 1 (explicitly states "screenplay by Lanthimos and Efthymis Filippou")

This example has a trap: Sentence 3 mentions that "Euripides" wrote the original play. But the question asks "who wrote the film The Killing of a Sacred Deer," and the answer should be the film's screenwriters, not the Greek playwright from thousands of years ago.

### Model results


| Model        | Finds correct answer? | Prediction                                      |
| ------------ | --------------------- | ----------------------------------------------- |
| Our Model    | ✓                     | Selected sentences 1 (correct) and 3            |
| XProvence v1 | ✗                     | Only selected sentence 3, missed correct answer |
| XProvence v2 | ✗                     | Only selected sentence 3, missed correct answer |


**Key Sentence Score Comparison:**

| Sentence                                     | Our Model | XProvence v1 | XProvence v2 |
| -------------------------------------------- | --------- | ------------ | ------------ |
| Sentence 1 (film screenplay, correct answer) | 0.915     | 0.133        | 0.081        |
| Sentence 3 (original play, distractor)       | 0.719     | 0.947        | 0.802        |


XProvence models:


*   Strongly attracted to "Euripides" and "play," giving sentence 3 near-perfect scores (0.947 and 0.802)
    
*   Completely ignores the actual answer (sentence 1), giving extremely low scores (0.133 and 0.081)
    
*   Even when lowering the threshold from 0.5 to 0.2, it still can't find the correct answer
    

Our model:

*   Correctly gives sentence 1 the highest score (0.915)
    
*   Still assigns sentence 3 some relevance (0.719) because it’s related to the background
    
*   Clearly separates the two with a ~0.2 margin
    

This example shows the model’s core strength: understanding **query intent** rather than just matching surface-level keywords. In this context, “Who wrote _The Killing of a Sacred Deer_” refers to the film, not the ancient Greek play. Our model picks up on that, while others get distracted by strong lexical cues.

## Try It Out and Tell Us What You Think

Our [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1) model is now fully open-sourced under the MIT license and ready for production use. You can plug it into your RAG pipeline, fine-tune it for your own domain, or build new tools on top of it. We also welcome contributions and feedback from the community.

*   **Download from HuggingFace**: [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1)
    
*   **All annotated training data**: [https://huggingface.co/zilliz/datasets](https://huggingface.co/zilliz/datasets) 
    

### Semantic Highlighting Available in Milvus and Zilliz Cloud

Semantic highlighting is also built directly into [Milvus](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/cloud) (the fully managed Milvus), giving users a clear view of _why_ each document was retrieved. Instead of scanning entire chunks, you immediately see the specific sentences that relate to your query — even when the wording doesn’t match exactly. This makes retrieval easier to understand and much faster to debug. For RAG pipelines, it also clarifies what the downstream LLM is expected to focus on, which helps with prompt design and quality checks.

[**Try Semantic Highlighting in a fully managed Zilliz Cloud for free**](https://cloud.zilliz.com/signup?utm_source=milvusio&utm_page=semantic-highlighting-blog) 

We’d love to hear how it works for you—bug reports, improvement ideas, or anything you discover while integrating it into your workflow.

If you want to talk through anything in more detail, feel free to join our [Discord channel](https://discord.com/invite/8uyFbECzPX) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session. We’re always happy to chat with other builders and swap notes.

## Acknowledgements

This work builds on a lot of great ideas and open-source contributions, and we want to highlight the projects that made this model possible.

*   **Provence** introduced a clean and practical framing for context pruning using lightweight encoder models.
    
*   **Open Provence** provided a solid, well-engineered codebase—training pipelines, data processing, and model heads—under a permissive license. It gave us a strong starting point for experimentation.
    

On top of that foundation, we added several contributions of our own:

*   Using **LLM reasoning** to generate higher-quality relevance labels
    
*   Creating **nearly 5 million** bilingual training samples aligned with real RAG workloads
    
*   Choosing a base model better suited for long-context relevance scoring (**BGE-M3 Reranker v2**)
    
*   Training only the **Pruning Head** to specialize the model for semantic highlighting
    

We’re grateful to the Provence and Open Provence teams for publishing their work openly. Their contributions significantly accelerated our development and made this project possible.


Semantic Highlighting for RAG Context Pruning and Token Saving


How We Built a Semantic Highlighting Model for RAG Context Pruning and Token Saving


LLM Context Pruning: Improving RAG and Agentic AI Systems


LLM Context Pruning: A Developer’s Guide to Better RAG and Agentic AI Results



_This post is written by Min Yin, one of the most active community contributors of Milvus, and is published here with permission._ 


[MinIO](https://github.com/minio/minio) is an open-source, high-performance, and S3-compatible object storage system widely used in AI/ML, analytics, and other data-intensive workloads. For many [Milvus](https://milvus.io/) deployments, it has also been the default choice for object storage. Recently, however, the MinIO team updated its [GitHub README](https://github.com/minio/minio?tab=readme-ov-file) to state that **_this project is no longer accepting new changes_**_._

![](https://assets.zilliz.com/minio_7b7df16860.png)

Actually, over the past few years, MinIO has gradually shifted its attention toward commercial offerings, tightened its licensing and distribution model, and scaled back active development in the community repository. Moving the open-source project into maintenance mode is the natural outcome of that broader transition.

For Milvus users who rely on MinIO by default, this change is hard to ignore. Object storage sits at the heart of Milvus’s persistence layer, and its reliability over time depends not just on what works today but on whether the system continues to evolve alongside the workloads it supports.

Against this backdrop, this article explores [RustFS](https://github.com/rustfs/rustfs) as a potential alternative. RustFS is a Rust-based, S3-compatible object storage system that emphasizes memory safety and modern systems design. It is still experimental, and this discussion is not a production recommendation. 


## The Milvus Architecture and Where the Object Storage Component Sits

Milvus 2.6 adopts a fully decoupled storage–compute architecture. In this model, the storage layer is composed of three independent components, each serving a distinct role.

Etcd stores metadata, Pulsar or Kafka handles streaming logs, and object storage—typically MinIO or an S3-compatible service—provides durable persistence for vector data and index files. Because storage and compute are separated, Milvus can scale compute nodes independently while relying on a shared, reliable storage backend.

![](https://assets.zilliz.com/milvus_architecture_fe897f1098.webp)

### The Role of Object Storage in Milvus

Object storage is the durable storage layer in Milvus. Raw vector data is persisted as binlogs, and index structures such as HNSW and IVF_FLAT are stored there as well. 

This design makes compute nodes stateless. Query Nodes do not store data locally; instead, they load segments and indexes from object storage on demand. As a result, nodes can scale up or down freely, recover quickly from failures, and support dynamic load balancing across the cluster without data rebalancing at the storage layer.

```
my-milvus-bucket/
├── files/                          # rootPath (default)
│   ├── insert_log/                 # insert binlogs
│   │   └── {Collection_ID}/
│   │       └── {Partition_ID}/
│   │           └── {Segment_ID}/
│   │               └── {Field_ID}/
│   │                   └── {Log_ID}     # Per-field binlog files
│   │
│   ├── delta_log/                  # Delete binlogs
│   │   └── {Collection_ID}/
│   │       └── {Partition_ID}/
│   │           └── {Segment_ID}/
│   │               └── {Log_ID}        
│   │
│   ├── stats_log/                  # Statistical data (e.g., Bloom filters)
│   │   └── {Collection_ID}/
│   │       └── {Partition_ID}/
│   │           └── {Segment_ID}/
│   │               └── {Field_ID}/
│   │                   └── {Log_ID}
│   │
│   └── index_files/                # Index files
│       └── {Build_ID}_{Index_Version}_{Segment_ID}_{Field_ID}/
│           ├── index_file_0
│           ├── index_file_1
│           └── ...
```

### Why Milvus Uses the S3 API

Rather than defining a custom storage protocol, Milvus uses the S3 API as its object storage interface. S3 has become the de facto standard for object storage: major cloud providers such as AWS S3, Alibaba Cloud OSS, and Tencent Cloud COS support it natively, while open-source systems like MinIO, Ceph RGW, SeaweedFS, and RustFS are fully compatible.

By standardizing on the S3 API, Milvus can rely on mature, well-tested Go SDKs instead of maintaining separate integrations for each storage backend. This abstraction also gives users deployment flexibility: MinIO for local development, cloud object storage in production, or Ceph and RustFS for private environments. As long as an S3-compatible endpoint is available, Milvus does not need to know—or care—what storage system is used underneath.

```
# Milvus configuration file milvus.yaml
minio:
 address: localhost
 port: 9000
 accessKeyID: minioadmin
 secretAccessKey: minioadmin
 useSSL: false
 bucketName: milvus-bucket
```

## Evaluating RustFS as an S3-Compatible Object Storage Backend for Milvus

### Project Overview

RustFS is a distributed object storage system written in Rust. It is currently in the alpha stage (version 1.0.0-alpha.68) and aims to combine the operational simplicity of MinIO with Rust’s strengths in memory safety and performance. More details are available on [GitHub](https://github.com/rustfs/rustfs).

RustFS is still under active development, and its distributed mode has not yet been officially released. As a result, RustFS is not recommended for production or mission-critical workloads at this stage.


### Architecture Design

RustFS follows a design that is conceptually similar to MinIO. An HTTP server exposes an S3-compatible API, while an Object Manager handles object metadata, and a Storage Engine is responsible for data block management. At the storage layer, RustFS relies on standard file systems such as XFS or ext4.

For its planned distributed mode, RustFS intends to use etcd for metadata coordination, with multiple RustFS nodes forming a cluster. This design aligns closely with common object storage architectures, making RustFS familiar to users with MinIO experience.

![](https://assets.zilliz.com/architecture_design_852f73b2c8.png)

### Compatibility with Milvus

Before considering RustFS as an alternative object storage backend, it is necessary to evaluate whether it meets Milvus’s core storage requirements.


|  **Milvus Requirement**  |           **S3 API**          |         **RustFS Support**         |
| :----------------------: | :---------------------------: | :--------------------------------: |
|  Vector data persistence |    `PutObject`, `GetObject`   |          ✅ Fully supported         |
|   Index file management  | `ListObjects`, `DeleteObject` |          ✅ Fully supported         |
| Segment merge operations |        Multipart Upload       |          ✅ Fully supported         |
|  Consistency guarantees  |    Strong read-after-write    | ✅ Strong consistency (single-node) |

Based on this evaluation, RustFS’s current S3 API implementation satisfies Milvus’s baseline functional requirements. This makes it suitable for practical testing in non-production environments.


## Hands-On: Replacing MinIO with RustFS in Milvus

The goal of this exercise is to replace the default MinIO object storage service and deploy Milvus 2.6.7 using RustFS as the object storage backend.


### Prerequisites

1. Docker and Docker Compose are installed (version ≥ 20.10), and the system can pull images and run containers normally.

2. A local directory is available for object data storage, such as `/volume/data/` (or a custom path).

3. The host port 9000 is open for external access, or an alternative port is configured accordingly.

4. The RustFS container runs as a non-root user (`rustfs`). Ensure the host data directory is owned by UID 10001.


### Step 1: Create the Data Directory and Set Permissions

```
# Create the project directory
mkdir -p milvus-rustfs && cd milvus-rustfs
# Create the data directory
mkdir -p volumes/{rustfs, etcd, milvus}
# Update permissions for the RustFS directory
sudo chown -R 10001:10001 volumes/rustfs
```

**Download the Official Docker Compose File**

```
wget https://github.com/milvus-io/milvus/releases/download/v2.6.7/milvus-standalone-docker-compose.yml -O docker-compose.yml
```

### Step 2: Modify the Object Storage Service

**Define the RustFS Service**

Note: Make sure the access key and secret key match the credentials configured in the Milvus service.

```
rustfs:
 container_name: milvus-rustfs
 image: registry.cn-hangzhou.aliyuncs.com/rustfs/rustfs: latest
 environment:
 RUSTFS_ACCESS_KEY: minioadmin
 RUSTFS_SECRET_KEY: minioadmin
 RUSTFS_CONSOLE_ENABLE: “true”
 RUSTFS_REGION: us-east-1
 # RUSTFS_SERVER_DOMAINS: localhost# Optional; not required for local deployments
 ports:
 - “9001:9001”
 - “9000:9000”
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/rustfs:/data
 command: >
 --address :9000
 --console-enable
 /data
 healthcheck:
 test: [“CMD”, “curl”, “-f”, “http://localhost:9000/health"]
 interval: 30s
 timeout: 20s
 retries: 3
```

**Complete Configuration**

Note: Milvus’s storage configuration currently assumes MinIO-style defaults and does not yet allow custom access key or secret key values. When using RustFS as a replacement, it must use the same default credentials expected by Milvus.

```
version: ‘3.5’
services:
 etcd:
 container_name: milvus-etcd
 image: registry.cn-hangzhou.aliyuncs.com/etcd/etcd: v3.5.25
 environment:
 - ETCD_AUTO_COMPACTION_MODE=revision
 - ETCD_AUTO_COMPACTION_RETENTION=1000
 - ETCD_QUOTA_BACKEND_BYTES=4294967296
 - ETCD_SNAPSHOT_COUNT=50000
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
 command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
 healthcheck:
 test: [“CMD”, “etcdctl”, “endpoint”, “health”]
 interval: 30s
 timeout: 20s
 retries: 3
 rustfs:
 container_name: milvus-rustfs
 image: registry.cn-hangzhou.aliyuncs.com/rustfs/rustfs: latest
 environment:
 RUSTFS_ACCESS_KEY: minioadmin
 RUSTFS_SECRET_KEY: minioadmin
 RUSTFS_CONSOLE_ENABLE: “true”
 RUSTFS_REGION: us-east-1
 # RUSTFS_SERVER_DOMAINS: localhost# Optional; not required for local deployments
 ports:
 - “9001:9001”
 - “9000:9000”
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/rustfs:/data
 command: >
 --address :9000
 --console-enable
 /data
 healthcheck:
 test: [“CMD”, “curl”, “-f”, “http://localhost:9000/health"]
 interval: 30s
 timeout: 20s
 retries: 3
 standalone:
 container_name: milvus-standalone
 image: registry.cn-hangzhou.aliyuncs.com/milvus/milvus: v2.6.7
 command: [”milvus“, ”run“, ”standalone“]
 security_opt:
 - seccomp: unconfined
 environment:
 MINIO_REGION: us-east-1
 ETCD_ENDPOINTS: etcd:2379
 MINIO_ADDRESS: rustfs:9000
 MINIO_ACCESS_KEY: minioadmin
 MINIO_SECRET_KEY: minioadmin
 MINIO_USE_SSL: ”false“
 MQ_TYPE: rocksmq
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
 healthcheck:
 test: [”CMD“, ”curl“, ”-f“, ”http://localhost:9091/healthz"]
 interval: 30s
 start_period: 90s
 timeout: 20s
 retries: 3
 ports:
 - “19530:19530”
 - “9091:9091”
 depends_on:
 - “etcd”
 - “rustfs”
networks:
 default:
 name: milvus
```

**Start the Services**

```
docker-compose -f docker-compose.yaml up -d
```

**Check Service Status**

```
docker-compose ps -a
```

![](https://assets.zilliz.com/code_d64dc88a96.png)

**Access the RustFS Web UI**

Open the RustFS web interface in your browser: <http://localhost:9001>

Log in using the default credentials: username and password are both minioadmin.

**Test the Milvus Service**

```
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
# connect to Milvus
connections.connect(
 alias="default",
 host='localhost',
 port='19530'
)
print("✓ Successfully connected to Milvus!")
# create test collection
fields = [
 FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
 FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128)
]
schema = CollectionSchema(fields=fields, description="test collection")
collection = Collection(name="test_collection", schema=schema)
print("✓ Test collection created!")
print("✓ RustFS verified as the S3 storage backend!")

### Step 3: Storage Performance Testing (Experimental)

**Test Design**

Two Milvus deployments were set up on identical hardware (16 cores / 32 GB memory / NVMe SSD), using RustFS and MinIO respectively as the object storage backend. The test dataset consisted of 1,000,000 vectors with 768 dimensions, using an HNSW index with parameters _M = 16_ and _efConstruction = 200_. Data was inserted in batches of 5,000.

The following metrics were evaluated: Insert throughput, Index build time, Cold and warm load time, Search latency, Storage footprint.

**Test Code**

Note: Only the core parts of the test code are shown below.

```
def milvus_load_bench(dim=768, rows=1_000_000, batch=5000):
    collection = Collection(...)
    # Insert test
    t0 = time.perf_counter()
    for i in range(0, rows, batch):
        collection.insert([rng.random((batch, dim), dtype=np.float32).tolist()])
    insert_time = time.perf_counter() - t0
    # Index build
    collection.flush()
    collection.create_index(field_name="embedding", 
                           index_params={"index_type": "HNSW", ...})
    # Load test (cold start + two warm starts)
    collection.release()
    load_times = []
    for i in range(3):
        if i > 0: collection.release(); time.sleep(2)
        collection.load()
        load_times.append(...)
    # Query test
    search_times = []
    for _ in range(3):
        collection.search(queries, limit=10, ...)
```

**Test Command**

```
# RustFS: --port 19530 --s3-endpoint http://localhost:9000 --s3-bucket bench
# MinIO: --port 19531 --s3-endpoint http://localhost:9001 --s3-bucket a-bucket
python bench.py milvus-load-bench --dim 768 --rows 1000000 --batch 5000 \
  --index-type HNSW --repeat-load 3 --release-before-load --do-search --drop-after
```

**Performance Results**

- **RustFS**

Faster writes (+57%), lower storage usage (–57%), and faster warm loads (+67%), making it suitable for write-heavy, cost-sensitive workloads. 

Much slower queries (7.96 ms vs. 1.85 ms, ~+330% latency) with noticeable variance (up to 17.14 ms), and 43% slower index builds. Not suitable for query-intensive applications.

- **MinIO**

Excellent query performance (**1.85 ms** average latency), mature small-file and random I/O optimizations, and a well-established ecosystem.


|     **Metric**    |  **RustFS**  |   **MinIO**  | **Difference** |
| :---------------: | :----------: | :----------: | :------------: |
| Insert Throughput | 4,472 rows/s | 2,845 rows/s |      0.57      |
|  Index Build Time |     803 s    |     562 s    |      -43%      |
| Load (Cold Start) |    22.7 s    |    18.3 s    |      -24%      |
| Load (Warm Start) |    0.009 s   |    0.027 s   |      0.67      |
|   Search Latency  |    7.96 ms   |    1.85 ms   |    **-330%**   |
|   Storage Usage   |    7.8 GB    |    18.0 GB   |      0.57      |

RustFS significantly outperforms MinIO in write performance and storage efficiency, with both showing roughly 57% improvement. This demonstrates the system-level advantages of the Rust ecosystem. However, the 330% gap in query latency limits RustFS’s suitability for query-intensive workloads.

For **production environments**, cloud-managed object storage services like **AWS S3** are recommended first, as they are mature, stable, and require no operational effort. Self-hosted solutions are better suited to specific scenarios: RustFS for cost-sensitive or write-intensive workloads, MinIO for query-intensive use cases, and Ceph for data sovereignty. With further optimization in random read performance, RustFS has the potential to become a strong self-hosted option.


## Conclusion

MinIO’s decision to stop accepting new community contributions is disappointing for many developers, but it won’t disrupt Milvus users. Milvus depends on the S3 API—not any specific vendor implementation—so swapping storage backends is straightforward. This S3-compatibility layer is intentional: it ensures Milvus stays flexible, portable, and decoupled from vendor lock-in.

For production deployments, cloud-managed services such as AWS S3 and Alibaba Cloud OSS remain the most reliable options. They’re mature, highly available, and drastically reduce the operational load compared to running your own object storage. Self-hosted systems like MinIO or Ceph still make sense in cost-sensitive environments or where data sovereignty is non-negotiable, but they require significantly more engineering overhead to operate safely at scale.

RustFS is interesting—Apache 2.0-licensed, Rust-based, and community-driven—but it's still early. The performance gap is noticeable, and the distributed mode hasn’t shipped yet. It’s not production-ready today, but it’s a project worth watching as it matures.

If you’re comparing object storage options for Milvus, evaluating MinIO replacements, or weighing the operational trade-offs of different backends, we’d love to hear from you.

Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) and share your thoughts. You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Evaluating RustFS for Milvus S3-Compatible Object Storage


MinIO Stops Accepting Community Changes: Evaluating RustFS as a Viable S3-Compatible Object Storage Backend for Milvus 



In high-quality AI agent systems, memory design is far more complex than it first appears. At its core, it must answer three fundamental questions: How should conversation history be stored? When should past context be retrieved? And what, exactly, should be retrieved?

These choices directly shape an agent’s response latency, resource usage, and—ultimately—its capability ceiling.

Models like ChatGPT and Claude feel increasingly “memory-aware” the more we use them. They remember preferences, adapt to long-term goals, and maintain continuity across sessions. In that sense, they already function as mini AI agents. Yet beneath the surface, their memory systems are built on very different architectural assumptions.

Recent reverse-engineering analyses of [ChatGPT](https://manthanguptaa.in/posts/chatgpt_memory/)’s and [Claude’s memory mechanisms](https://manthanguptaa.in/posts/claude_memory/) reveal a clear contrast. **ChatGPT** relies on precomputed context injection and layered caching to deliver lightweight, predictable continuity. **Claude,** by contrast, adopts RAG-style, on-demand retrieval with dynamic memory updates to balance memory depth and efficiency.

These two approaches are not just design preferences—they are shaped by infrastructure capability. [**Milvus 2.6**](https://milvus.io/docs/release_notes.md#v268) introduces the combination of hybrid dense–sparse retrieval, efficient scalar filtering, and tiered storage that on-demand conversational memory requires, making selective retrieval fast and economical enough to deploy in real-world systems.

In this post, we’ll walk through how ChatGPT’s and Claude’s memory systems actually work, why they diverged architecturally, and how recent advances in systems like Milvus make on-demand conversational retrieval practical at scale.


## ChatGPT’s Memory System

Instead of querying a vector database or dynamically retrieving past conversations at inference time, ChatGPT constructs its “memory” by assembling a fixed set of context components and injecting them directly into every prompt. Each component is prepared ahead of time and occupies a known position in the prompt.

This design keeps personalization and conversational continuity intact while making latency, token usage, and system behavior more predictable. In other words, memory is not something the model searches for on the fly—it is something the system packages and hands to the model every time it generates a response.

At a high level, a complete ChatGPT prompt consists of the following layers, ordered from most global to most immediate:

[0] System Instructions

[1] Developer Instructions

[2] Session Metadata (ephemeral)

[3] User Memory (long-term facts)

[4] Recent Conversations Summary (past chats, titles + snippets)

[5] Current Session Messages (this chat)

[6] Your latest message

Among these, components [2] through [5] form the effective memory of the system, each serving a distinct role.


### Session Metadata

Session metadata represents short-lived, non-persistent information that is injected once at the beginning of a conversation and discarded when the session ends. Its role is to help the model adapt to the current usage context rather than to personalize behavior long term. 

This layer captures signals about the user’s immediate environment and recent usage patterns. Typical signals include:

- **Device information** — for example, whether the user is on mobile or desktop

- **Account attributes** — such as subscription tier (for example, ChatGPT Go), account age, and overall usage frequency

- **Behavioral metrics** — including active days over the past 1, 7, and 30 days, average conversation length, and model usage distribution (for example, 49% of requests handled by GPT-5)


### User Memory

User memory is the persistent, editable layer of memory that enables personalization across conversations. It stores relatively stable information—such as a user’s name, role or career goals, ongoing projects, past outcomes, and learning preferences—and is injected into each new conversation to preserve continuity over time.

This memory can be updated in two ways:

- **Explicit updates** occur when users directly manage memory with instructions like “remember this” or “remove this from memory.”

- **Implicit updates** occur when the system identifies information that meets OpenAI’s storage criteria—such as a confirmed name or job title—and saves it automatically, subject to the user’s default consent and memory settings.


### Recent Conversation Summary

The recent conversation summary is a lightweight, cross-session context layer that preserves continuity without replaying or retrieving full chat histories. Instead of relying on dynamic retrieval, as in traditional RAG-based approaches, this summary is precomputed and injected directly into every new conversation.

This layer summarizes user messages only, excluding assistant replies. It is intentionally limited in size—typically around 15 entries—and retains only high-level signals about recent interests rather than detailed content. Because it does not rely on embeddings or similarity search, it keeps both latency and token consumption low.


### Current Session Messages

Current session messages contain the full message history of the ongoing conversation and provide the short-term context needed for coherent, turn-by-turn responses. This layer includes both user inputs and assistant replies, but only while the session remains active.

Because the model operates within a fixed token limit, this history cannot grow indefinitely. When the limit is reached, the system drops the earliest messages to make room for newer ones. This truncation affects only the current session: long-term user memory and the recent conversation summary remain intact.


## Claude’s Memory System

Claude takes a different approach to memory management. Rather than injecting a large, fixed bundle of memory components into every prompt—as ChatGPT does—Claude combines persistent user memory with on-demand tools and selective retrieval. Historical context is fetched only when the model judges it to be relevant, allowing the system to trade off contextual depth against computational cost.

Claude’s prompt context is structured as follows:

[0] System Prompt (static instructions)

[1] User Memories

[2] Conversation History

[3] Current Message

The key differences between Claude and ChatGPT lie in **how conversation history is retrieved** and **how user memory is updated and maintained**.


### User Memories

In Claude, user memories form a long-term context layer similar in purpose to ChatGPT’s user memory, but with a stronger emphasis on automatic, background-driven updates. These memories are stored in a structured format (wrapped in XML-style tags) and are designed to evolve gradually over time with minimal user intervention.

Claude supports two update paths:

- **Implicit updates** — The system periodically analyzes conversation content and updates memory in the background. These updates are not applied in real time, and memories associated with deleted conversations are gradually pruned as part of ongoing optimization.

- **Explicit updates** — Users can directly manage memory through commands such as “remember this” or “delete this,” which are executed via a dedicated `memory_user_edits` tool.

Compared with ChatGPT, Claude places greater responsibility on the system itself to refine, update, and prune long-term memory. This reduces the need for users to actively curate what is stored.


### Conversation History

For conversation history, Claude does not rely on a fixed summary that is injected into every prompt. Instead, it retrieves past context only when the model decides it is necessary, using three distinct mechanisms. This avoids carrying irrelevant history forward and keeps token usage under control.

|               **Component**               |                                                                     **Purpose**                                                                     |                                                              **How It’s Used**                                                             |
| :---------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: |
| **Rolling Window (Current Conversation)** |                  Stores the full message history of the current conversation (not a summary), similar to ChatGPT’s session context                  |                     Injected automatically. Token limit is ~190K; older messages are dropped once the limit is reached                    |
|       `conversation_search` **tool**      |              Searches past conversations by topic or keyword, returning conversation links, titles, and user/assistant message excerpts             | Triggered when the model determines that historical details are needed. Parameters include `query` (search terms) and `max_results` (1–10) |
|          `recent_chats` **tool**          | Retrieves recent conversations within a specified time range (for example, “past 3 days”), with results formatted the same as `conversation_search` |    Triggered when the recent, time-scoped context is relevant. Parameters include `n` (number of results), `sort_order`, and time range    |

Among these components, `conversation_search` is especially noteworthy. It can surface relevant results even for loosely phrased or multilingual queries, indicating that it operates at a semantic level rather than relying on simple keyword matching. This likely involves embedding-based retrieval, or a hybrid approach that first translates or normalizes the query into a canonical form and then applies keyword or hybrid retrieval.

Overall, Claude’s on-demand retrieval approach has several notable strengths:

- **Retrieval is not automatic**: Tool calls are triggered by the model’s own judgment. For example, when a user refers to _“the project we discussed last time,”_ Claude may decide to invoke `conversation_search` to retrieve the relevant context.

- **Richer context when needed**: Retrieved results can include **assistant response excerpts**, whereas ChatGPT’s summaries only capture user messages. This makes Claude better suited for use cases that require deeper or more precise conversational context.

- **Better efficiency by default**: Because historical context is not injected unless needed, the system avoids carrying large amounts of irrelevant history forward, reducing unnecessary token consumption.

The trade-offs are equally clear. Introducing on-demand retrieval increases system complexity: indexes must be built and maintained, queries executed, results ranked, and sometimes re-ranked. End-to-end latency also becomes less predictable than with precomputed, always-injected context. In addition, the model must learn to decide when retrieval is necessary. If that judgment fails, relevant context may never be fetched at all.


## The Constraints Behind Claude-Style On-Demand Retrieval

Adopting an on-demand retrieval model makes the vector database a critical part of the architecture. Conversation retrieval places unusually high demands on both storage and query execution, and the system must meet four constraints at the same time.

![](https://assets.zilliz.com/constraints_b6ed74e454.jpg)

### 1. Low Latency Tolerance

In conversational systems, P99 latency typically needs to stay under ~20 ms. Delays beyond that are immediately noticeable to users. This leaves little room for inefficiency: vector search, metadata filtering, and result ranking must all be carefully optimized. A bottleneck at any point can degrade the entire conversational experience.


### 2. Hybrid Search Requirement

User queries often span multiple dimensions. A request like _“discussions about RAG from the past week”_ combines semantic relevance with time-based filtering. If a database only supports vector search, it may return 1,000 semantically similar results, only for application-layer filtering to reduce them to a handful—wasting most of the computation. To be practical, the database must natively support combined vector and scalar queries.


### 3. Storage–Compute Separation

Conversation history exhibits a clear hot–cold access pattern. Recent conversations are queried frequently, while older ones are rarely touched. If all vectors had to stay in memory, storing tens of millions of conversations would consume hundreds of gigabytes of RAM—an impractical cost at scale. To be viable, the system must support storage–compute separation, keeping hot data in memory and cold data in object storage, with vectors loaded on demand.


### 4. Diverse Query Patterns

Conversation retrieval does not follow a single access pattern. Some queries are purely semantic (for example, _“the performance optimization we discussed”_), others are purely temporal (_“all conversations from last week”_), and many combine multiple constraints (_“Python-related discussions mentioning FastAPI in the last three months”_). The database query planner must adapt execution strategies to different query types, rather than relying on a one-size-fits-all, brute-force search.

Together, these four challenges define the core constraints of conversational retrieval. Any system seeking to implement Claude-style, on-demand retrieval must address all of them in a coordinated way.


## Why Milvus 2.6 Works Well for Conversational Retrieval

The design choices in [Milvus 2.6](https://milvus.io/docs/release_notes.md#v268) align closely with the core requirements of on-demand conversational retrieval. Below is a breakdown of the key capabilities and how they map to real conversational retrieval needs.

![](https://assets.zilliz.com/milvus_2_6_ce379ff42d.jpg)

### Hybrid Retrieval with Dense and Sparse Vectors

Milvus 2.6 natively supports storing dense and sparse vectors within the same collection and automatically fusing their results at query time. Dense vectors (for example, 768-dimensional embeddings generated by models like BGE-M3) capture semantic similarity, while sparse vectors (typically produced by BM25) preserve exact keyword signals. 

For a query such as _“discussions about RAG from last week,”_ Milvus executes semantic retrieval and keyword retrieval in parallel, then merges the results through reranking. Compared to using either approach alone, this hybrid strategy delivers significantly higher recall in real conversational scenarios.


### Storage–Compute Separation and Query Optimization

Milvus 2.6 supports tiered storage in two ways:

- Hot data in memory, cold data in object storage

- Indexes in memory, raw vector data in object storage

With this design, storing one million conversation entries can be achieved with roughly 2 GB of memory and 8 GB of object storage. With proper tuning, P99 latency can remain below 20 ms, even with storage–compute separation enabled.


### JSON Shredding and Fast Scalar Filtering

Milvus 2.6 enables JSON Shredding by default, flattening nested JSON fields into columnar storage. This improves scalar filtering performance by 3–5× according to official benchmarks (actual gains vary by query pattern).

Conversational retrieval often requires filtering by metadata such as user ID, session ID, or time range. With JSON Shredding, queries like _“all conversations from user A in the past week”_ can be executed directly on columnar indexes, without repeatedly parsing full JSON blobs.


### Open-Source Control and Operational Flexibility

As an open-source system, Milvus offers a level of architectural and operational control that closed, black-box solutions do not. Teams can tune index parameters, apply data tiering strategies, and customize distributed deployments to match their workloads.

This flexibility lowers the barrier to entry: small and mid-sized teams can build million- to tens-of-millions-scale conversational retrieval systems without relying on oversized infrastructure budgets.


## Why ChatGPT and Claude Took Different Paths

At a high level, the difference between ChatGPT’s and Claude’s memory systems comes down to how each handles forgetting. ChatGPT favors proactive forgetting: once memory exceeds fixed limits, older context is dropped. This trades completeness for simplicity and predictable system behavior. Claude favors delayed forgetting. In theory, conversation history can grow without bound, with recall delegated to an on-demand retrieval system.

So why did the two systems choose different paths? With the technical constraints laid out above, the answer becomes clear: **each architecture is only viable if the underlying infrastructure can support it**.

If Claude’s approach had been attempted in 2020, it would likely have been impractical. At the time, vector databases often incurred hundreds of milliseconds of latency, hybrid queries were poorly supported, and resource usage scaled prohibitively as data grew. Under those conditions, on-demand retrieval would have been dismissed as over-engineering.

By 2025, the landscape has changed. Advances in infrastructure—driven by systems such as **Milvus 2.6**—have made storage–compute separation, query optimization, dense–sparse hybrid retrieval, and JSON Shredding viable in production. These advances reduce latency, control costs, and make selective retrieval practical at scale. As a result, on-demand tools and retrieval-based memory have become not only feasible, but increasingly attractive, especially as a foundation for agent-style systems.

Ultimately, architecture choices follow what the infrastructure makes possible.


## Conclusion

In real-world systems, memory design is not a binary choice between precomputed context and on-demand retrieval. The most effective architectures are typically hybrid, combining both approaches.

A common pattern is to inject recent conversation turns through a sliding context window, store stable user preferences as fixed memory, and retrieve older history on demand via vector search. As a product matures, this balance can shift gradually—from primarily precomputed context to increasingly retrieval-driven—without requiring a disruptive architectural reset.

Even when starting with a precomputed approach, it is important to design with migration in mind. Memory should be stored with clear identifiers, timestamps, categories, and source references. When retrieval becomes viable, embeddings can be generated for existing memory and added to a vector database alongside the same metadata, allowing retrieval logic to be introduced incrementally and with minimal disruption.

Have questions or want a deep dive on any feature of the latest Milvus? Join our [Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on [GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Milvus 2.6 Makes Claude-Style On-Demand Retrieval Practical


Reflections on ChatGPT and Claude’s Memory Systems: What It Takes to Enable On-Demand Conversational Retrieval


Open-sourcing a Bilingual Semantic Highlighting Model for Production AI


We Trained and Open-Sourced a Bilingual Semantic Highlighting Model for Production RAG and AI Search



When building a database system, engineers spend most of their time on performance: index types, recall, latency, throughput, and scaling. But once a system moves beyond a single developer’s laptop, another question becomes just as critical: **who can do what inside your Milvus cluster**? In other words—access control.

Across the industry, many operational incidents stem from simple permission mistakes. A script runs against the wrong environment. A service account has broader access than intended. A shared admin credential ends up in CI. These issues usually surface as very practical questions:

- Are developers allowed to delete production collections?

- Why can a test account read production vector data?

- Why are multiple services logging in with the same admin role?

- Can analytics jobs have read-only access with zero write privileges?

[Milvus](https://milvus.io/) addresses these challenges with [role-based access control (RBAC)](https://milvus.io/docs/rbac.md). Instead of giving every user superadmin rights or trying to enforce restrictions in application code, RBAC lets you define precise permissions at the database layer. Each user or service gets exactly the capabilities it needs—nothing more.

This post explains how RBAC works in Milvus, how to configure it, and how to apply it safely in production environments.


## Why Access Control Matters When Using Milvus 

When teams are small, and their AI applications serve only a limited number of users, infrastructure is usually simple. A few engineers manage the system; Milvus is used only for development or testing; and operational workflows are straightforward. In this early stage, access control rarely feels urgent—because the risk surface is small and any mistakes can be easily reversed.

As Milvus moves into production and the number of users, services, and operators grows, the usage model changes quickly. Common scenarios include:

- Multiple business systems sharing the same Milvus instance

- Multiple teams accessing the same vector collections

- Test, staging, and production data coexisting in a single cluster

- Different roles needing different levels of access, from read-only queries to writes and operational control

Without well-defined access boundaries, these setups create predictable risks:

- Test workflows might accidentally delete production collections

- Developers might unintentionally modify indexes used by live services

- Widespread use of the `root` account makes actions impossible to trace or audit

- A compromised application might gain unrestricted access to all vector data

As usage grows, relying on informal conventions or shared admin accounts is no longer sustainable. A consistent, enforceable access model becomes essential—and this is exactly what Milvus RBAC provides.


## What is RBAC in Milvus

[RBAC (Role-Based Access Control)](https://milvus.io/docs/rbac.md) is a permission model that controls access based on **roles** rather than individual users. In Milvus, RBAC lets you define exactly which operations a user or service is allowed to perform—and on which specific resources. It provides a structured, scalable way to manage security as your system grows from a single developer to a complete production environment.

Milvus RBAC is built around the following core components:

![Users Roles Privileges](https://assets.zilliz.com/users_roles_privileges_030620f913.png)

- **Resource**: The entity being accessed. In Milvus, resources include the **instance**, **database**, and **collection**.

- **Privilege**: A specific allowed operation on a resource—for example, creating a collection, inserting data, or deleting entities. 

- **Privilege Group**: A predefined set of related privileges, such as “read-only” or “write.”

- **Role**: A combination of privileges and the resources they apply to. A role determines _what_ operations can be performed and _where_.

- **User**: An identity in Milvus. Each user has a unique ID and is assigned one or more roles.

These components form a clear hierarchy:

1. **Users are assigned roles**

2. **Roles define privileges**

3. **Privileges apply to specific resources**

A key design principle in Milvus is that **permissions are never assigned directly to users**. All access goes through roles. This indirection simplifies administration, reduces configuration errors, and makes permission changes predictable.

This model scales cleanly in real deployments. When multiple users share a role, updating the role’s privileges instantly updates permissions for all of them—without modifying each user individually. It’s a single point of control aligned with how modern infrastructure manages access.


## How RBAC Works in Milvus

When a client sends a request to Milvus, the system evaluates it through a series of authorization steps. Each step must pass before the operation is allowed to proceed:

![How RBAC Works in Milvus](https://assets.zilliz.com/how_rbac_works_afe48bc717.png)

1. **Authenticate the request:** Milvus first verifies the user identity. If authentication fails, the request is rejected with an authentication error. 

2. **Check role assignment:** After authentication, Milvus checks whether the user has at least one role assigned. If no role is found, the request is rejected with a permission denied error.

3. **Verify required privileges:** Milvus then evaluates whether the user’s role grants the required privilege on the target resource. If the privilege check fails, the request is rejected with a permission denied error.

4. **Execute the operation:** If all checks pass, Milvus executes the requested operation and returns the result.


## How to Configure Access Control via RBAC in Milvus

### 1. Prerequisites

Before RBAC rules can be evaluated and enforced, user authentication must be enabled so that every request to Milvus can be associated with a specific user identity.

Here are two standard deployment methods.

- **Deploying with Docker Compose**

If Milvus is deployed using Docker Compose, edit the `milvus.yaml` configuration file and enable authorization by setting `common.security.authorizationEnabled` to `true`:

```
common:
  security:
    authorizationEnabled: true
```

- **Deploying with Helm Charts**

If Milvus is deployed using Helm Charts, edit the `values.yaml` file and add the following configuration under `extraConfigFiles.user.yaml`:

```
extraConfigFiles:
  user.yaml: |+
    common:
      security:
        authorizationEnabled: true
```

### 2. Initialization

By default, Milvus creates a built-in `root` user when the system starts. The default password for this user is `Milvus`. 

As an initial security step, use the `root` user to connect to Milvus and change the default password immediately. It is strongly recommended to use a complex password to prevent unauthorized access.

```
from pymilvus import MilvusClient
# Connect to Milvus using the default root user
client = MilvusClient(
    uri='http://localhost:19530', 
    token="root:Milvus"
)
# Update the root password
client.update_password(
    user_name="root",
    old_password="Milvus", 
    new_password="xgOoLudt3Kc#Pq68"
)
```

### 3. Core Operations

**Create Users**

For daily usage, it is recommended to create dedicated users instead of using the `root` account. 


```
client.create_user(user_name="user_1", password="P@ssw0rd")
```

**Create Roles**

Milvus provides a built-in `admin` role with full administrative privileges. For most production scenarios, however, it is recommended to create custom roles to achieve finer-grained access control.


```
client.create_role(role_name="role_a")
```

**Create Privilege Groups**

A privilege group is a collection of multiple privileges. To simplify permission management, related privileges can be grouped and granted together.

Milvus includes the following built-in privilege groups:

- `COLL_RO`, `COLL_RW`, `COLL_ADMIN`

- `DB_RO`, `DB_RW`, `DB_ADMIN`

- `Cluster_RO`, `Cluster_RW`, `Cluster_ADMIN`

Using these built-in privilege groups can significantly reduce the complexity of permission design and improve consistency across roles.

You can either use the built-in privilege groups directly or create custom privilege groups as needed.

```
# Create a privilege group
client.create_privilege_group(group_name='privilege_group_1'）
# Add privileges to the privilege group
client.add_privileges_to_group(group_name='privilege_group_1', privileges=['Query', 'Search'])
```

**Grant Privileges or Privilege Groups to Roles**

After a role is created, privileges or privilege groups can be granted to the role. The target resources for these privileges can be specified at different levels, including the instance, database, or individual Collections.

```
client.grant_privilege_v2(
    role_name="role_a",
    privilege="Search",
    collection_name='collection_01',
    db_name='default',
)
client.grant_privilege_v2(
    role_name="role_a",
    privilege="privilege_group_1",
    collection_name='collection_01',
    db_name='default',
)
client.grant_privilege_v2(
    role_name="role_a",
    privilege="ClusterReadOnly",
    collection_name='*',
    db_name='*',
)
```

**Grant Roles to Users**

Once roles are assigned to a user, the user can access resources and perform the operations defined by those roles. A single user can be granted one or multiple roles, depending on the required access scope.

```
client.grant_role(user_name="user_1", role_name="role_a")
```

### 4. Inspect and Revoke Access

**Inspect Roles Assigned to a User**

```
client.describe_user(user_name="user_1")
```

**Inspect Privileges Assigned to a Role**

```
client.describe_role(role_name="role_a")
```

**Revoke Privileges from a Role**

```
client.revoke_privilege_v2(
    role_name="role_a",
    privilege="Search",
    collection_name='collection_01',
    db_name='default',
)
client.revoke_privilege_v2(
    role_name="role_a",
    privilege="privilege_group_1",
    collection_name='collection_01',
    db_name='default',
)
```

**Revoke Roles from a User**

```
client.revoke_role(
    user_name='user_1',
    role_name='role_a'
)
```

**Delete Users and Roles**

```
client.drop_user(user_name="user_1")
client.drop_role(role_name="role_a")
```

## Example: Access Control Design for a Milvus-Powered RAG System

Consider a Retrieval-Augmented Generation (RAG) system built on top of Milvus.

In this system, different components and users have clearly separated responsibilities, and each requires a different level of access.

| Actor                    | Responsibility                      | Required Access               |
| ------------------------ | ----------------------------------- | ----------------------------- |
| Platform Administrator   | System operations and configuration | Instance-level administration |
| Vector Ingestion Service | Vector data ingestion and updates   | Read and write access         |
| Search Service           | Vector search and retrieval         | Read-only access              |

```
from pymilvus import MilvusClient
client = MilvusClient(
    uri='http://localhost:19530',
    token="root:xxx"  # Replace with the updated root password
)
# 1. Create a user (use a strong password)
client.create_user(user_name="rag_admin", password="xxx")
client.create_user(user_name="rag_reader", password="xxx")
client.create_user(user_name="rag_writer", password="xxx")
# 2. Create roles
client.create_role(role_name="role_admin")
client.create_role(role_name="role_read_only")
client.create_role(role_name="role_read_write")
# 3. Grant privileges to the role
## Using built-in Milvus privilege groups
client.grant_privilege_v2(
    role_name="role_admin",
    privilege="Cluster_Admin",
    collection_name='*',
    db_name='*',
)
client.grant_privilege_v2(
    role_name="role_read_only",
    privilege="COLL_RO",
    collection_name='*',
    db_name='default',
)
client.grant_privilege_v2(
    role_name="role_read_write",
    privilege="COLL_RW",
    collection_name='*',
    db_name='default',
)
# 4. Assign the role to the user
client.grant_role(user_name="rag_admin", role_name="role_admin")
client.grant_role(user_name="rag_reader", role_name="role_read_only")
client.grant_role(user_name="rag_writer", role_name="role_read_write")
```

## Quick Tips: How to Operate Access Control Safely in Production

To ensure access control remains effective and manageable in long-running production systems, follow these practical guidelines.

**1. Change the default** `root` **password and limit the use of** `root` **account**

Update the default `root` password immediately after initialization and restrict its use to administrative tasks only. Avoid using or sharing the root account for routine operations. Instead, create dedicated users and roles for day-to-day access to reduce risk and improve accountability.

**2. Physically isolate Milvus instances across environments**

Deploy separate Milvus instances for development, staging, and production. Physical isolation provides a stronger safety boundary than logical access control alone and significantly reduces the risk of cross-environment mistakes.

**3. Follow the principle of least privilege**

Grant only the permissions required for each role:

- **Development environments:** permissions can be more permissive to support iteration and testing

- **Production environments:** permissions should be strictly limited to what is necessary

- **Regular audits:** periodically review existing permissions to ensure they are still required

**4. Actively revoke permissions when they are no longer needed**

Access control is not a one-time setup—it requires ongoing maintenance. Revoke roles and privileges promptly when users, services, or responsibilities change. This prevents unused permissions from accumulating over time and becoming hidden security risks.


## Conclusion

Configuring access control in Milvus is not inherently complex, but it is essential for operating the system safely and reliably in production. With a well-designed RBAC model, you can:

- **Reduce risk** by preventing accidental or destructive operations

- **Improve security** by enforcing least-privilege access to vector data

- **Standardize operations** through a clear separation of responsibilities

- **Scale with confidence**, laying the foundation for multi-tenant and large-scale deployments

Access control is not an optional feature or a one-time task. It is a foundational part of operating Milvus safely over the long term.

👉 Start building a solid security baseline with [RBAC](https://milvus.io/docs/rbac.md) for your Milvus deployment.

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Milvus RBAC Guide: How to Control Access to Your Vector Database


Milvus RBAC Explained: Secure Your Vector Database with Role-Based Access Control 



As unstructured data continues to explode and AI models keep getting smarter, vector search has become the default retrieval layer for many AI systems—RAG pipelines, AI search, agents, recommendation engines, and more. It works because it captures meaning: not just the words users type, but the intent behind them. 

Once these applications move into production, however, teams often discover that semantic understanding is only one side of the retrieval problem. Many workloads also depend on strict textual rules—such as matching exact terminology, preserving word order, or identifying phrases that carry technical, legal, or operational significance.

[Milvus 2.6](https://milvus.io/docs/release_notes.md#v267) removes that split by introducing native full-text search directly into the vector database. With token and positional indexes built into the core engine, Milvus can interpret a query’s semantic intent while enforcing precise keyword and phrase-level constraints. The result is a unified retrieval pipeline in which meaning and structure reinforce each other rather than living in separate systems.

[Phrase Match](https://milvus.io/docs/phrase-match.md) is a key part of this full-text capability. It identifies sequences of terms that appear together and in order—crucial for detecting log patterns, error signatures, product names, and any text in which word order defines meaning. In this post, we’ll explain how [Phrase Match](https://milvus.io/docs/phrase-match.md) works in [Milvus](https://milvus.io/), how `slop` adds flexibility needed for real-world text, and why these features make hybrid vector–full-text search not just possible but practical within a single database.


## What is Phrase Match? 

Phrase Match is a full-text query type in Milvus that focuses on _structure_—specifically, whether a sequence of words appears in the same order inside a document. When no flexibility is allowed, the query behaves strictly: the terms must appear next to each other and in sequence. A query like **"robotics machine learning"** therefore matches only when those three words occur as a continuous phrase. 

The challenge is that real text rarely behaves this neatly. Natural language introduces noise: extra adjectives slip in, logs reorder fields, product names gain modifiers, and human authors don’t write with query engines in mind. A strict phrase match breaks easily—one inserted word, one rephrasing, or one swapped term can cause a miss. And in many AI systems, especially production-facing ones, missing a relevant log line or rule-triggering phrase isn’t acceptable.

Milvus 2.6 addresses this friction with a simple mechanism: **slop**. Slop defines _the amount of wiggle room allowed between query_ terms. Instead of treating a phrase as brittle and inflexible, slop lets you decide whether one extra word is tolerable, or two, or even whether slight reordering should still count as a match. This moves phrase search from a binary pass–fail test to a controlled, tunable retrieval tool.

To see why this matters, imagine searching logs for all variants of the familiar networking error **"connection reset by peer."** In practice, your logs might look like:

```
connection reset by peer
connection fast reset by peer
connection was suddenly reset by the peer
peer reset connection by ...
peer unexpected connection reset happened
```

At a glance, all of these represent the same underlying event. But common retrieval methods struggle:


### BM25 struggles with structure.

It views the query as a bag of keywords, ignoring the order in which they appear. As long as “connection” and “peer” show up somewhere, BM25 may rank the document highly—even if the phrase is reversed or unrelated to the concept you're actually searching for.


### Vector search struggles with constraints.

Embeddings excel at capturing meaning and semantic relationships, but they cannot enforce a rule like “these words must appear in this sequence.” You might retrieve semantically related messages, but still miss the exact structural pattern required for debugging or compliance.

Phrase Match fills the gap between these two approaches. By using **slop**, you can specify exactly how much variation is acceptable: 

- `slop = 0` — Exact match (All terms must appear contiguously and in order.)

- `slop = 1` — Allow one extra word (Covers common natural-language variations with a single inserted term.)

- `slop = 2` — Allow multiple inserted words (Handles more descriptive or verbose phrasing.)

- `slop = 3` — Allow reordering (Supports reversed or loosely ordered phrases, often the hardest case in real-world text.)

Instead of hoping the scoring algorithm “gets it right,” you explicitly declare the structural tolerance your application requires.


## How Phrase Match Works in Milvus 

Powered by the [Tantivy](https://github.com/quickwit-oss/tantivy) search engine library, Phrase Match in Milvus is implemented on top of an inverted index with positional information. Instead of only checking whether terms appear in a document, it verifies that they appear in the right order and within a controllable distance. 

The diagram below illustrates the process:

![](https://assets.zilliz.com/phrase_match_workflow_a4f3badb66.png)

**1. Document Tokenization (with Positions)**

When documents are inserted into Milvus, text fields are processed by an [analyzer](https://milvus.io/docs/analyzer-overview.md), which splits the text into tokens (words or terms) and records each token’s position within the document. For example, `doc_1` is tokenized as: `machine (pos=0), learning (pos=1), boosts (pos=2), efficiency (pos=3)`. 

**2. Inverted Index Creation**

Next, Milvus builds an inverted index. Instead of mapping documents to their contents, the inverted index maps each token to the documents in which it appears, along with all recorded positions of that token within each document.

**3. Phrase Matching**

When a phrase query is executed, Milvus first uses the inverted index to identify documents that contain all query tokens. It then validates each candidate by comparing token positions to ensure the terms appear in the correct order and within the allowed `slop` distance. Only documents that satisfy both conditions are returned as matches.

The diagram below summarizes how Phrase Match works end-to-end.

![](https://assets.zilliz.com/workflow2_63c168b107.png)

## How to Enable Phrase Match in Milvus

Phrase Match works on fields of type **`VARCHAR`**, the string type in Milvus. To use it, you must configure your collection schema so that Milvus performs text analysis and stores positional information for the field. This is done by enabling two parameters: `enable_analyzer` and `enable_match`.


### Set enable_analyzer and enable_match

To turn on Phrase Match for a specific VARCHAR field, set both parameters to `True` when defining the field schema. Together, they tell Milvus to:

- **tokenize** the text (via `enable_analyzer`), and

- **build an inverted index with positional offsets** (via `enable_match`).

Phrase Match relies on both steps: the analyzer breaks text into tokens, and the match index stores where those tokens appear, enabling efficient phrase and slop-based queries.

Below is an example schema configuration that enables Phrase Match on a `text` field:

```
from pymilvus import MilvusClient, DataType

schema = MilvusClient.create_schema(enable_dynamic_field=False)
schema.add_field(
    field_name="id",
    datatype=DataType.INT64,
    is_primary=True,
    auto_id=True
)
schema.add_field(
    field_name='text',                 # Name of the field
    datatype=DataType.VARCHAR,         # Field data type set as VARCHAR (string)
    max_length=1000,                   # Maximum length of the string
    enable_analyzer=True,              # Enables text analysis (tokenization)
    enable_match=True                  # Enables inverted indexing for phrase matching
)
schema.add_field(
    field_name="embeddings",
    datatype=DataType.FLOAT_VECTOR,
    dim=5
)
```

## Search with Phrase Match: How Slop Affects the Candidate Set

Once you’ve enabled match for a VARCHAR field in your collection schema, you can perform phrase matches using the `PHRASE_MATCH` expression.

Note: The `PHRASE_MATCH` expression is case-insensitive. You can use either `PHRASE_MATCH` or `phrase_match`.

In search operations, Phrase Match is commonly applied before vector similarity ranking. It first filters documents based on explicit textual constraints, narrowing the candidate set. The remaining documents are then re-ranked using vector embeddings.

The example below shows how different `slop` values affect this process. By adjusting the `slop` parameter, you directly control which documents pass the phrase filter and proceed to the vector ranking stage.

Suppose you have a collection named `tech_articles` containing the following five entities:


| **doc_id** | **text**                                                          |
| ----------- | ----------------------------------------------------------------- |
| 1           | Machine learning boosts efficiency in large-scale data analysis   |
| 2           | Learning a machine-based approach is vital for modern AI progress |
| 3           | Deep learning machine architectures optimize computational loads  |
| 4           | Machine swiftly improves model performance for ongoing learning   |
| 5           | Learning advanced machine algorithms expands AI capabilities      |

**`slop=1`**

Here, we allow a slop of 1. The filter is applied to documents that contain the phrase “learning machine” with slight flexibility.

```
# Example: Filter documents containing "learning machine" with slop=1
filter_slop1 = "PHRASE_MATCH(text, 'learning machine', 1)"

result_slop1 = client.search(
    collection_name="tech_articles",
    anns_field="embeddings",
    data=[query_vector],
    filter=filter_slop1,
    search_params={"params": {"nprobe": 10}},
    limit=10,
    output_fields=["id", "text"]
)
```

Match results:

| doc_id | text                                                              |
| ------- | ----------------------------------------------------------------- |
| 2       | Learning a machine-based approach is vital for modern AI progress |
| 3       | Deep learning machine architectures optimize computational loads  |
| 5       | Learning advanced machine algorithms expands AI capabilities      |

**`slop=2`**

This example allows a slop of 2, meaning that up to two extra tokens (or reversed terms) are allowed between the words “machine” and “learning”.

```
# Example: Filter documents containing "machine learning" with slop=2
filter_slop2 = "PHRASE_MATCH(text, 'machine learning', 2)"

result_slop2 = client.search(
    collection_name="tech_articles",
    anns_field="embeddings",             # Vector field name
    data=[query_vector],                 # Query vector
    filter=filter_slop2,                 # Filter expression
    search_params={"params": {"nprobe": 10}},
    limit=10,                            # Maximum results to return
    output_fields=["id", "text"]
)
```

Match results:

| **doc_id** |                             **text**                             |
| :---------: | :--------------------------------------------------------------: |
|      1      |  Machine learning boosts efficiency in large-scale data analysis |
|      3      | Deep learning machine architectures optimize computational loads |

**`slop=3`**

In this example, a slop of 3 provides even more flexibility. The filter searches for “machine learning” with up to three token positions allowed between the words.

```
# Example: Filter documents containing "machine learning" with slop=3
filter_slop3 = "PHRASE_MATCH(text, 'machine learning', 3)"

result_slop2 = client.search(
    collection_name="tech_articles",
    anns_field="embeddings",             # Vector field name
    data=[query_vector],                 # Query vector
    filter=filter_slop3,                 # Filter expression
    search_params={"params": {"nprobe": 10}},
    limit=10,                            # Maximum results to return
    output_fields=["id", "text"]
)
```

Match results:

| **doc_id** |                              **text**                             |
| :---------: | :---------------------------------------------------------------: |
|      1      |  Machine learning boosts efficiency in large-scale data analysis  |
|      2      | Learning a machine-based approach is vital for modern AI progress |
|      3      |  Deep learning machine architectures optimize computational loads |
|      5      |    Learning advanced machine algorithms expands AI capabilities   |


## Quick Tips: What You Need to Know Before Enabling Phrase Match in Milvus

Phrase Match provides support for phrase-level filtering, but enabling it involves more than query-time configuration. It is helpful to be aware of the associated considerations before applying it in a production setting.

- Enabling Phrase Match on a field creates an inverted index, which increases storage usage. The exact cost depends on factors such as text length, the number of unique tokens, and the analyzer configuration. When working with large text fields or high-cardinality data, this overhead should be considered upfront.

- Analyzer configuration is another critical design choice. Once an analyzer is defined in the collection schema, it cannot be changed. Switching to a different analyzer later requires dropping the existing collection and recreating it with a new schema. For this reason, analyzer selection should be treated as a long-term decision rather than an experiment.

- Phrase Match behavior is tightly coupled to how text is tokenized. Before applying an analyzer to an entire collection, it is recommended to use the `run_analyzer` method to inspect the tokenization output and confirm that it matches your expectations. This step can help avoid subtle mismatches and unexpected query results later. For more information, refer to [Analyzer Overview](https://milvus.io/docs/analyzer-overview.md#share-DYZvdQ2vUowWEwx1MEHcdjNNnqT).


## Conclusion

Phrase Match is a core full-text search type that enables phrase-level and positional constraints beyond simple keyword matching. By operating on token order and proximity, it provides a predictable and precise way to filter documents based on how terms actually appear in text.

In modern retrieval systems, Phrase Match is commonly applied before vector-based ranking. It first restricts the candidate set to documents that explicitly satisfy required phrases or structures. Vector search is then used to rank these results by semantic relevance. This pattern is especially effective in scenarios such as log analysis, technical documentation search, and RAG pipelines, where textual constraints must be enforced before semantic similarity is considered.

With the introduction of the `slop` parameter in Milvus 2.6, Phrase Match becomes more tolerant of natural language variation while retaining its role as a full-text filtering mechanism. This makes phrase-level constraints easier to apply in production retrieval workflows.

👉 Try it out with the [demo](https://github.com/openvino-book/Milvus-Phrase-Match-Demo) scripts, and explore [Milvus 2.6](https://milvus.io/docs/release_notes.md#v267) to see how phrase-aware retrieval fits into your stack.

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Phrase Match with Slop: Better Full-Text Search Accuracy in Milvus


Phrase Match with Slop in Milvus 2.6: How to Improve Phrase-Level Full-Text Search Accuracy 


How to Safely Upgrade from Milvus 2.5.x to Milvus 2.6.x



[Retrieval Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation) has become the default approach for providing context and memory for AI applications — AI agents, customer support assistants, knowledge bases, and search systems all rely on it. 

In almost every RAG pipeline, the standard process is the same: take the documents, split them into chunks, and then embed those chunks for similarity retrieval in a vector database like [Milvus](https://milvus.io/). Because **chunking** happens upfront, the quality of those chunks directly affects how well the system retrieves information and how accurate the final answers are.

The issue is that traditional chunking strategies usually split text without any semantic understanding. Fixed-length chunking cuts based on token counts, and recursive chunking uses surface-level structure, but both still ignore the actual meaning of the text. As a result, related ideas often get separated, unrelated lines get grouped together, and important context gets fragmented.

In this blog, I’d like to share a different chunking strategy: [**Max–Min Semantic Chunking**](https://link.springer.com/article/10.1007/s10791-025-09638-7). Instead of chunking first, it embeds the text upfront and uses semantic similarity to decide where boundaries should form. By embedding before cutting, the pipeline can track natural shifts in meaning rather than relying on arbitrary length limits.

## How a Typical RAG Pipeline Works

Most RAG pipelines, regardless of the framework, follow the same four-stage assembly line. You’ve probably written some version of this yourself:


### 1. Data Cleaning and Chunking

The pipeline starts by cleaning the raw documents: removing headers, footers, navigation text, and anything that isn’t real content. Once the noise is out, the text gets split into smaller pieces. Most teams use fixed-size chunks — typically 300–800 tokens — because it keeps the embedding model manageable. The downside is that the splits are based on length, not meaning, so the boundaries can be arbitrary.


### 2. Embedding and Storage

Each chunk is then embedded using an embedding model like OpenAI’s [`text-embedding-3-small`](https://zilliz.com/ai-models/text-embedding-3-small) or BAAI’s encoder. The resulting vectors are stored in a vector database such as [Milvus](https://milvus.io/) or [Zilliz Cloud](https://zilliz.com/cloud). The database handles indexing and similarity search so you can quickly compare new queries against all stored chunks.


### 3. Querying

When a user asks a question — for example, _“How does RAG reduce hallucinations?”_ — the system embeds the query and sends it to the database. The database returns the top-K chunks whose vectors are closest to the query. These are the pieces of text the model will rely on to answer the question.


### 4. Answer Generation

The retrieved chunks are bundled together with the user query and fed into an LLM. The model generates an answer using the provided context as grounding.

**Chunking sits at the start of this whole pipeline, but it has an outsized impact**. If the chunks align with the natural meaning of the text, retrieval feels accurate and consistent. If the chunks were cut in awkward places, the system has a harder time finding the right information, even with strong embeddings and a fast vector database.


### The Challenges of Getting Chunking Right

Most RAG systems today use one of two basic chunking methods, both of which have limitations.

**1. Fixed-size chunking**

This is the simplest approach: split the text by a fixed token or character count. It’s fast and predictable, but completely unaware of grammar, topics, or transitions. Sentences can get cut in half. Sometimes even words. The embeddings you get from these chunks tend to be noisy because the boundaries don’t reflect how the text is actually structured.

**2. Recursive character splitting**

This method is a bit smarter. It splits text hierarchically based on cues like paragraphs, line breaks, or sentences. If a section is too long, it recursively divides it further. The output is generally more coherent, but still inconsistent. Some documents lack clear structure or have uneven section lengths, which hurts retrieval accuracy. And in some cases, this approach still produces chunks that exceed the model’s context window.

Both methods face the same tradeoff: precision vs. context. Smaller chunks improve retrieval accuracy but lose surrounding context; larger chunks preserve meaning but risk adding irrelevant noise. Striking the right balance is what makes chunking both foundational—and frustrating—in RAG system design.


## Max–Min Semantic Chunking: Embed First, Chunk Later

In 2025, S.R. Bhat et al. published [_Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Analysis_](https://arxiv.org/abs/2505.21700). One of their key findings was that there isn’t a single **“best”** chunk size for RAG. Small chunks (64–128 tokens) tend to work better for factual or lookup-style questions, while larger chunks (512–1024 tokens) help with narrative or high-level reasoning tasks. In other words, fixed-size chunking is always a compromise.

This raises a natural question: instead of picking one length and hoping for the best, can we chunk by meaning rather than size? [**Max–Min Semantic Chunking**](https://link.springer.com/article/10.1007/s10791-025-09638-7) is one approach I found that tries to do precisely that.

The idea is simple: **embed first, chunk second**. Instead of splitting text and then embedding whatever pieces fall out, the algorithm embeds _all sentences_ up front. It then uses the semantic relationships between those sentence embeddings to decide where the boundaries should go.

![Diagram showing embed-first chunk-second workflow in Max-Min Semantic Chunking](https://assets.zilliz.com/embed_first_chunk_second_94f69c664c.png)

Conceptually, the method treats chunking as a constrained clustering problem in embedding space. You walk through the document in order, one sentence at a time. For each sentence, the algorithm compares its embedding with those in the current chunk. If the new sentence is semantically close enough, it joins the chunk. If it’s too far, the algorithm starts a new chunk. The key constraint is that chunks must follow the original sentence order — no reordering, no global clustering.

The result is a set of variable-length chunks that reflect where the document’s meaning actually changes, instead of where a character counter happens to hit zero.


## How the Max–Min Semantic Chunking Strategy Works

Max–Min Semantic Chunking determines chunk boundaries by comparing how sentences relate to one another in the high-dimensional vector space. Instead of relying on fixed lengths, it looks at how meaning shifts across the document. The process can be broken down into six steps:


### 1. Embed all sentences and start a chunk

The embedding model converts each sentence in the document into a vector embedding. It processes sentences in order. If the first _n–k_ sentences form the current chunk C, the following sentence (sₙ₋ₖ₊₁) needs to be evaluated: should it join C, or start a new chunk?


### 2. Measure how consistent the current chunk is

Within chunk C, calculate the minimum pairwise cosine similarity among all sentence embeddings. This value reflects how closely related the sentences within the chunk are. A lower minimum similarity indicates that the sentences are less related, suggesting that the chunk may need to be split.


### 3. Compare the new sentence to the chunk

Next, calculate the maximum cosine similarity between the new sentence and any sentence already in C. This reflects how well the new sentence aligns semantically with the existing chunk.


### 4. Decide whether to extend the chunk or start a new one

This is the core rule:

- If the **new sentence’s max similarity** to chunk **C** is **greater than or equal to the** **minimum similarity inside C**, → The new sentence fits and stays in the chunk.

- Otherwise, → start a new chunk.

This ensures that each chunk maintains its internal semantic consistency.


### 5. Adjust thresholds as the document changes

To optimize chunk quality, parameters such as chunk size and similarity thresholds can be adjusted dynamically. This allows the algorithm to adapt to varying document structures and semantic densities.


### 6. Handle the first few sentences

When a chunk contains only one sentence, the algorithm handles the first comparison using a fixed similarity threshold. If the similarity between sentence 1 and sentence 2 is above that threshold, they form a chunk. If not, they split immediately.


## Strengths and Limitations of Max–Min Semantic Chunking

Max–Min Semantic Chunking improves how RAG systems split text by using meaning instead of length, but it’s not a silver bullet. Here’s a practical look at what it does well and where it still falls short.


### What It Does Well

Max–Min Semantic Chunking improves on traditional chunking in three important ways:


#### **1. Dynamic, meaning-driven chunk boundaries**

Unlike fixed-size or structure-based approaches, this method relies on semantic similarity to guide chunking. It compares the minimum similarity within the current chunk (how cohesive it is) to the maximum similarity between the new sentence and that chunk (how well it fits). If the latter is higher, the sentence joins the chunk; otherwise, a new chunk starts.


#### **2. Simple, practical parameter tuning**

The algorithm depends on just three core hyperparameters:

- the **maximum chunk size**,

- the **minimum similarity** between the first two sentences, and

- the **similarity threshold** for adding new sentences.

These parameters adjust automatically with context—larger chunks require stricter similarity thresholds to maintain coherence.


#### **3. Low processing overhead**

Because the RAG pipeline already computes sentence embeddings, Max–Min Semantic Chunking doesn’t add heavy computation. All it needs is a set of cosine similarity checks while scanning through sentences. This makes it cheaper than many semantic chunking techniques that require extra models or multi-stage clustering.


### What It Still Can’t Solve

Max–Min Semantic Chunking improves chunk boundaries, but it doesn’t eliminate all the challenges of document segmentation. Because the algorithm processes sentences in order and only clusters locally, it can still miss long-range relationships in longer or more complex documents.

One common issue is **context fragmentation**. When important information is spread across different parts of a document, the algorithm may place those pieces into separate chunks. Each chunk then carries only part of the meaning. 

For example, in the Milvus 2.4.13 Release Notes, as shown below, one chunk might contain the version identifier while another contains the feature list. A query like _“What new features were introduced in Milvus 2.4.13?”_ depends on both. If those details are split across different chunks, the embedding model may not connect them, leading to weaker retrieval.

+ ![Example showing context fragmentation in Milvus 2.4.13 Release Notes with version identifier and feature list in separate chunks](https://assets.zilliz.com/v2413_a98e1b1f99.png)

This fragmentation also affects the LLM generation stage. If the version reference is in one chunk and the feature descriptions are in another, the model receives incomplete context and can’t reason cleanly about the relationship between the two.

To mitigate these cases, systems often use techniques such as sliding windows, overlapping chunk boundaries, or multi-pass scans. These approaches reintroduce some of the missing context, reduce fragmentation, and help the retrieval step retain related information.


## Conclusion

Max–Min Semantic Chunking isn’t a magic fix for every RAG problem, but it does give us a more sane way to think about chunk boundaries. Instead of letting token limits decide where ideas get chopped, it uses embeddings to detect where the meaning actually shifts. For many real-world documents—APIs, specs, logs, release notes, troubleshooting guides—this alone can push retrieval quality noticeably higher.

What I like about this approach is that it fits naturally into existing RAG pipelines. If you already embed sentences or paragraphs, the extra cost is basically a few cosine similarity checks. You don’t need extra models, complex clustering, or heavyweight preprocessing. And when it works, the chunks it produces feel more “human”—closer to how we mentally group information when reading.

But the method still has blind spots. It only sees meaning locally, and it can’t reconnect information that’s intentionally spread apart. Overlapping windows, multi-pass scans, and other context-preserving tricks are still necessary, especially for documents where references and explanations live far from each other.

Still, Max–Min Semantic Chunking moves us in the right direction: away from arbitrary text slicing and toward retrieval pipelines that actually respect semantics. If you’re exploring ways to make RAG more reliable, it’s worth experimenting with.

Have questions or want to dig deeper into improving RAG performance? Join our [Discord](https://discord.com/invite/8uyFbECzPX) and connect with engineers who are building and tuning real retrieval systems every day.


Max–Min Semantic Chunking: Top Chunking Strategy to Improve RAG Performance


Embedding First, Then Chunking: Smarter RAG Retrieval with Max–Min Semantic Chunking 



If you’ve worked with long-running LLM conversations, you’ve probably had this frustrating moment: halfway through a long thread, the model starts drifting. Answers get vague, reasoning weakens, and key details mysteriously vanish. But if you drop the exact same prompt into a new chat, suddenly the model behaves—focused, accurate, grounded.

This isn’t the model “getting tired” — it’s **context rot**. As a conversation grows, the model has to juggle more information, and its ability to prioritize slowly declines. [Antropic studies](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) show that as context windows stretch from around 8K tokens to 128K, retrieval accuracy can drop by 15–30%. The model still has room, but it loses track of what matters. Bigger context windows help delay the problem, but they don’t eliminate it.

This is where **context engineering** comes in. Instead of handing the model everything at once, we shape what it sees: retrieving only the pieces that matter, compressing what no longer needs to be verbose, and keeping prompts and tools clean enough for the model to reason over. The goal is simple: make important information available at the right moment, and ignore the rest.

Retrieval plays a central role here, especially for long-running agents. Vector databases like [**Milvus**](https://milvus.io/) provide the foundation for efficiently pulling relevant knowledge back into context, letting the system stay grounded even as tasks grow in depth and complexity.

In this blog, we’ll look at how context rot happens, the strategies teams use to manage it, and the architectural patterns — from retrieval to prompt design — that keep AI agents sharp across long, multi-step workflows.


## Why Context Rot Happens 

People often assume that giving an AI model more context naturally leads to better answers. But that’s not actually true. Humans struggle with long inputs too: cognitive science suggests our working memory holds roughly **7±2 chunks** of information. Push beyond that, and we start to forget, blur, or misinterpret details.

LLMs show similar behavior—just at a much larger scale and with more dramatic failure modes.

The root issue comes from the [Transformer architecture](https://zilliz.com/learn/decoding-transformer-models-a-study-of-their-architecture-and-underlying-principles) itself. Every token must compare itself against every other token, forming pairwise attention across the entire sequence. That means computation grows **O(n²)** with context length. Expanding your prompt from 1K tokens to 100K doesn’t make the model “work harder”—it multiplies the number of token interactions by **10,000×**.

![](https://assets.zilliz.com/contextual_dilution_622033db72.png)

**Then there’s the problem with the training data.** Models see far more short sequences than long ones. So when you ask an LLM to operate across extremely large contexts, you’re pushing it into a regime it wasn’t heavily trained for. In practice, very long-context reasoning is often **out of distribution** for most models.

Despite these limits, long context is now unavoidable. Early LLM applications were mostly single-turn tasks—classification, summarization, or simple generation. Today, more than 70% of enterprise AI systems rely on agents that stay active across many rounds of interaction, often for hours, managing branching, multi-step workflows. Long-lived sessions have moved from exception to default.

Then the next question is: **how do we keep the model’s attention sharp without overwhelming it?** 


## Context Retrieval Approaches to Solving Context Rot

Retrieval is one of the most effective levers we have to combat context rot, and in practice it tends to show up in complementary patterns that address context rot from different angles.


### 1. Just-in-Time Retrieval: Reducing Unnecessary Context

One major cause of context rot is _overloading_ the model with information it doesn’t need yet. Claude Code—Anthropic’s coding assistant—solves this issue with **Just-in-Time (JIT) retrieval**, a strategy where the model fetches information only when it becomes relevant.

Instead of stuffing entire codebases or datasets into its context (which greatly increases the chance of drift and forgetting), Claude Code maintains a tiny index: file paths, commands, and documentation links. When the model needs a piece of information, it retrieves that specific item and inserts it into context **at the moment it matters**—not before. 

For example, if you ask Claude Code to analyze a 10GB database, it never tries to load the whole thing. It works more like an engineer:

1. Runs a SQL query to pull high-level summaries of the dataset.

2. Uses commands like `head` and `tail` to view sample data and understand its structure.

3. Retains only the most important information—such as key statistics or sample rows—within the context.

By minimizing what’s kept in context, JIT retrieval prevents the buildup of irrelevant tokens that cause rot. The model stays focused because it only ever sees the information required for the current reasoning step.


### 2. Pre-retrieval (Vector Search): Preventing Context Drift Before It Starts

Sometimes the model can’t “ask” for information dynamically—customer support, Q&A systems, and agent workflows often need the right knowledge available _before_ generation begins. This is where **pre-retrieval** becomes critical.

Context rot often happens because the model is handed a large pile of raw text and expected to sort out what matters. Pre-retrieval flips that: a vector database (like [Milvus](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/cloud)) identifies the most relevant pieces _before_ inference, ensuring only high-value context reaches the model.

In a typical RAG setup:

- Documents are embedded and stored in a vector database, such as Milvus.

- At query time, the system retrieves a small set of highly relevant chunks through similarity searches.

- Only those chunks go into the model’s context.

This prevents rot in two ways: 

- **Noise reduction:** irrelevant or weakly related text never enters the context in the first place.

- **Efficiency:** models process far fewer tokens, reducing the chance of losing track of essential details.

Milvus can search millions of documents in milliseconds, making this approach ideal for live systems where latency matters. 


### 3. Hybrid JIT and Vector Retrieval

Vector search-based pre-retrieval addresses a significant part of context rot by ensuring the model starts with high-signal information rather than raw, oversized text. But Anthropic highlights two real challenges that teams often overlook:

- **Timeliness:** If the knowledge base updates faster than the vector index is rebuilt, the model may rely on stale information.

- **Accuracy:** Before a task begins, it’s hard to predict precisely what the model will need—especially for multi-step or exploratory workflows.

So in real world workloads, a hybrid appaorch is the optimal solution. 

- Vector search for stable, high-confidence knowledge

- Agent-driven JIT exploration for information that evolves or only becomes relevant mid-task

By blending these two approaches, you get the speed and efficiency of vector retrieval for known information, and the flexibility for the model to discover and load new data whenever it becomes relevant. 

Let’s look at how this works in a real system. Take a production documentation assistant, for example. Most teams eventually settle on a two-stage pipeline: Milvus-powered vector search + agent-based JIT retrieval. 

**1. Milvus Powered Vector Search (Pre-retrieval)** 

- Convert your documentation, API references, changelogs, and known issues into embeddings.

- Store them in the Milvus Vector Database with metadata like product area, version, and update time.

- When a user asks a question, run a semantic search to grab the top-K relevant segments.

This resolves roughly 80% of routine queries in under 500 ms, giving the model a strong, context-rot-resistant starting point.

**2. Agent-Based Exploration**

When the initial retrieval isn’t sufficient—e.g., when the user asks for something highly specific or time-sensitive—the agent can call tools to fetch new information:

- Use `search_code` to locate specific functions or files in the codebase

- Use `run_query` to pull real-time data from the database

- Use `fetch_api` to obtain the latest system status

These calls typically take **3–5 seconds**, but they ensure the model always works with fresh, accurate, and relevant data—even for questions the system couldn’t anticipate beforehand.

This hybrid structure ensures context remains timely, correct, and task-specific, dramatically reducing the risk of context rot in long-running agent workflows.

Milvus is especially effective in these hybrid scenarios because it supports:

- **Vector search + scalar filtering**, combining semantic relevance with structured constraints

- **Incremental updates**, allowing embeddings to be refreshed without downtime

This makes Milvus an ideal backbone for systems that need both semantic understanding and precise control over what gets retrieved.

![](https://assets.zilliz.com/milvus_in_hybrid_architecture_7d4e391aa4.png)

For example, you might run a query like:
```
# You can combine queries like this in Milvus
collection.search(
    data=[query_embedding],  # Semantic similarity
    anns_field="embedding",
    param={"metric_type": "COSINE", "params": {"nprobe": 10}},
    expr="doc_type == 'API' and update_time > '2025-01-01'",  # Structured filtering
    limit=5
)
```

## How to Choose the Right Approach for Dealing With Context Rot

With vector-search pre-retrieval, Just-in-Time retrieval, and hybrid retrieval all available, the natural question is: **which one should you use?**

Here is a simple but practical way to choose—based on how _stable_ your knowledge is and how _predictable_ the model’s information needs are.


### 1. Vector Search → Best for Stable Domains

If the domain changes slowly but demands precision—finance, legal work, compliance, medical documentation—then a Milvus-powered knowledge base with **pre-retrieval** is usually the right fit.

The information is well-defined, updates are infrequent, and most questions can be answered by retrieving semantically relevant documents upfront.

**Predictable tasks + stable knowledge → Pre-retrieval.**


### 2. Just-in-Time Retrieval → Best for Dynamic, Exploratory Workflows

Fields like software engineering, debugging, analytics, and data science involve rapidly changing environments: new files, new data, new deployment states. The model can’t predict what it will need before the task starts. 

**Unpredictable tasks + fast-changing knowledge → Just-in-Time retrieval.**


### 3. Hybrid Approach → When Both Conditions Are True

Many real systems aren’t purely stable or purely dynamic. For example, developer documentation changes slowly, whereas the state of a production environment changes minute by minute. A hybrid approach lets you:

- Load known, stable knowledge using vector search (fast, low-latency)

- Fetch dynamic information with agent tools on demand (accurate, up-to-date)

**Mixed knowledge + mixed task structure → Hybrid retrieval approach.** 


## What if the Context Window Still Isn’t Enough

Context engineering helps reduce overload, but sometimes the problem is more fundamental: **the task simply won’t fit**, even with careful trimming. 

Certain workflows—like migrating a large codebase, reviewing multi-repository architectures, or generating deep research reports—can exceed 200K+ context windows before the model reaches the end of the task. Even with vector search doing heavy lifting, some tasks require more persistent, structured memory.

Recently, Anthropic has offered three practical strategies. 


### 1. Compression: Preserve Signal, Drop Noise

When the context window approaches its limit, the model can **compress earlier interactions** into concise summaries. Good compression keeps 

- Key decisions

- Constraints and requirements

- Outstanding issues

- Relevant samples or examples

And removes:

- Verbose tool outputs

- Irrelevant logs

- Redundant steps

The challenge is balance. Compress too aggressively, and the model loses critical information; compress too lightly, and you gain little space. Effective compression keeps the “why” and “what” while discarding the “how we got here.”


### 2. Structured Note-Taking: Move Stable Information Outside Context

Instead of keeping everything inside the model’s window, the system can store important facts in **external memory**—a separate database or a structured store that the agent can query as needed.

For example, Claude’s Pokémon-agent prototype stores durable facts like:

- `Pikachu leveled up to 8`

- `Trained 1234 steps on Route 1`

- `Goal: reach level 10`

Meanwhile, transient details—battle logs, long tool outputs—stay outside the active context. This mirrors how humans use notebooks: we don’t keep every detail in our working memory; we store reference points externally and look them up when needed.

Structured note-taking prevents context rot caused by repeated, unnecessary details while giving the model a reliable source of truth.


### 3. Sub-Agent Architecture: Divide and Conquer Large Tasks

For complex tasks, a multi-agent architecture can be designed where a lead agent oversees the overall work, while several specialized sub-agents handle specific aspects of the task. These sub-agents dive deep into large amounts of data related to their sub-tasks but only return the concise, essential results. This approach is commonly used in scenarios like research reports or data analysis.

![](https://assets.zilliz.com/longduration_task_cbbc07b9ca.png)

In practice, it's best to start by using a single agent combined with compression to handle the task. External storage should only be introduced when there’s a need to retain memory across sessions. The multi-agent architecture should be reserved for tasks that genuinely require parallel processing of complex, specialized sub-tasks.

Each approach extends the system's effective “working memory” without blowing the context window—and without triggering context rot.


## Best Practices for Designing Context That Actually Works

After handling context overflow, there’s another equally important piece: how the context is built in the first place. Even with compression, external notes, and sub-agents, the system will struggle if the prompt and tools themselves aren’t designed to support long, complex reasoning.

![](https://assets.zilliz.com/System_Prompts_cf655dcd0d.png)

Anthropic offers a helpful way to think about this—less as a single prompt-writing exercise, and more as constructing context across three layers.

### **System Prompts: Find the Goldilocks Zone**

Most system prompts fail at the extremes. Too much detail—lists of rules, nested conditions, hard-coded exceptions—makes the prompt brittle and difficult to maintain. Too little structure leaves the model guessing what to do.

The best prompts sit in the middle: structured enough to guide behavior, flexible enough for the model to reason. In practice, this means giving the model a clear role, a general workflow, and light tool guidance—nothing more, nothing less.

For example:

```
You are a technical documentation assistant serving developers.
1. Start by retrieving relevant documents from the Milvus knowledge base.  
2. If the retrieval results are insufficient, use the `search_code` tool to perform a deeper search in the codebase.  
3. When answering, cite specific documentation sections or code line numbers.

## Tool guidance
- search_docs: Used for semantic retrieval, best for conceptual questions.  
- search_code: Used for precise lookup in the codebase, best for implementation-detail questions.  
…
```

This prompt sets direction without overwhelming the model or forcing it to juggle dynamic information that doesn’t belong here.


### Tool Design: Less Is More

Once the system prompt sets the high-level behavior, tools carry the actual operational logic. A surprisingly common failure mode in tool-augmented systems is simply having too many tools—or having tools whose purposes overlap.

A good rule of thumb:

- **One tool, one purpose**

- **Explicit, unambiguous parameters**

- **No overlapping responsibilities**

If a human engineer would hesitate about which tool to use, the model will too. Clean tool design reduces ambiguity, lowers cognitive load, and prevents context from being cluttered with unnecessary tool attempts.

![](https://assets.zilliz.com/tooling_complexity_7d2bb60c54.png)

### Dynamic Information Should Be Retrieved, Not Hardcoded

The final layer is the easiest to overlook. Dynamic or time-sensitive information—such as status values, recent updates, or user-specific state—should not appear in the system prompt at all. Baking it into the prompt guarantees it will become stale, bloated, or contradictory over long tasks.

Instead, this information should be fetched only when needed, either through retrieval or via agent tools. Keeping dynamic content out of the system prompt prevents context rot and keeps the model’s reasoning space clean.


## Conclusion

As AI agents move into production environments across different industries, they’re taking on longer workflows and more complex tasks than ever before. In these settings, managing context becomes a practical necessity.

**However, a bigger context window doesn’t automatically produce better results**; in many cases, it does the opposite. When a model is overloaded, fed stale information, or forced through massive prompts, accuracy quietly drifts. That slow, subtle decline is what we now call **context rot**. 

Techniques like JIT retrieval, pre-retrieval, hybrid pipelines, and vector-database-powered semantic search all aim at the same goal: **making sure the model sees the right information at the right moment — no more, no less — so it can stay grounded and produce reliable answers.**

As an open-source, high-performance vector database, [**Milvus**](https://milvus.io/) sits at the core of this workflow. It provides the infrastructure for storing knowledge efficiently and retrieving the most relevant pieces with low latency. Paired with JIT retrieval and other complementary strategies, Milvus helps AI agents remain accurate as their tasks become deeper and more dynamic. 

But retrieval is only one piece of the puzzle. Good prompt design, a clean and minimal toolset, and sensible overflow strategies — whether compression, structured notes, or sub-agents — all work together to keep the model focused across long-running sessions. This is what real context engineering looks like: not clever hacks, but thoughtful architecture.

If you want AI agents that stay accurate over hours, days, or entire workflows, context deserves the same attention you’d give to any other core part of your stack. 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Context Engineering Strategies to Prevent LLM Context Rot with Milvus


Keeping AI Agents Grounded: Context Engineering Strategies that Prevent Context Rot Using Milvus 


How to Build AI Search Pipelines That Understand User Intent with Milvus


Building AI Search Pipelines That Understand What Users Want with Milvus 


LangChain 1.0 and Milvus: Build Production-Ready AI Agents with Long-Term Memory


LangChain 1.0 and Milvus: How to Build Production-Ready Agents with Real Long-Term Memory


Milvus Ngram Index: Faster Keyword Matching and LIKE Queries for Agent Workloads


Introducing the Milvus Ngram Index: Faster Keyword Matching and LIKE Queries for Agent Workloads


Milvus Tiered Storage: 80% Less Vector Search Cost with On-Demand Hot–Cold Data Loading


Stop Paying for Cold Data: 80% Cost Reduction with On-Demand Hot–Cold Data Loading in Milvus Tiered Storage



As AI systems move from experiments to production infrastructure, vector databases are no longer dealing with millions of embeddings. **Billions are now routine, and tens of billions are increasingly common.** At this scale, algorithmic choices affect not only performance and recall, but also translate directly into infrastructure cost.

This leads to a core question for large-scale deployments: **how do you choose the right index to deliver acceptable recall and latency without letting compute resource usage spiral out of control?**

Graph-based indexes such as **NSW, HNSW, CAGRA, and Vamana** have become the most widely adopted answer. By navigating pre-built neighborhood graphs, these indexes enable fast nearest-neighbor search at billion scale, avoiding brute-force scanning and comparison of every vector against the query.

However, the cost profile of this approach is uneven. **Querying a graph is relatively cheap; constructing it is not.** Building a high-quality graph requires large-scale distance computations and iterative refinement across the entire dataset—workloads that traditional CPU resources struggle to handle efficiently as data grows.

NVIDIA’s CAGRA addresses this bottleneck by using GPUs to accelerate graph construction through massive parallelism. While this significantly reduces build time, relying on GPUs for both index construction and query serving introduces higher cost and scalability constraints in production environments.

To balance these tradeoffs, [Milvus 2.6.1](https://milvus.io/docs/release_notes.md#v261) **adopts a hybrid design for** [GPU_CAGRA](https://milvus.io/docs/gpu-cagra.md) **indexes**: **GPUs are used only for graph construction, while query execution runs on CPUs.** This preserves the quality advantages of GPU-built graphs while keeping query serving scalable and cost-efficient—making it especially well suited for workloads with infrequent data updates, large query volumes, and strict cost sensitivity.


## What Is CAGRA and How Does It Work?

Graph-based vector indexes generally fall into two major categories:

- **Iterative graph construction**, represented by **CAGRA** (already supported in Milvus).

- **Insert-based graph construction**, represented by **Vamana** (currently under development in Milvus).

These two approaches differ significantly in their design goals and technical foundations, making each suitable for different data scales and workload patterns.

**NVIDIA CAGRA (CUDA ANN Graph-based)** is a GPU-native algorithm for approximate nearest neighbor (ANN) search, designed for building and querying large-scale proximity graphs efficiently. By leveraging GPU parallelism, CAGRA significantly accelerates graph construction and delivers high-throughput query performance compared with CPU-based approaches such as HNSW.

CAGRA is built on the **NN-Descent (Nearest Neighbor Descent)** algorithm, which constructs a k-nearest-neighbor (kNN) graph through iterative refinement. In each iteration, candidate neighbors are evaluated and updated, gradually converging toward higher-quality neighborhood relationships across the dataset.

After each refinement round, CAGRA applies additional graph pruning techniques—such as **2-hop detour pruning**—to remove redundant edges while preserving search quality. This combination of iterative refinement and pruning results in a **compact yet well-connected graph** that is efficient to traverse at query time.

Through repeated refinement and pruning, CAGRA produces a graph structure that supports **high recall and low-latency nearest-neighbor search at large scale**, making it particularly well suited for static or infrequently updated datasets.


### Step 1: Building the Initial Graph with NN-Descent

NN-Descent is based on a simple but powerful observation: if node _u_ is a neighbor of _v_, and node _w_ is a neighbor of _u_, then _w_ is very likely a neighbor of _v_ as well. This transitive property allows the algorithm to discover true nearest neighbors efficiently, without exhaustively comparing every pair of vectors.

CAGRA uses NN-Descent as its core graph construction algorithm. The process works as follows:

**1. Random initialization:** Each node starts with a small set of randomly selected neighbors, forming a rough initial graph.

**2. Neighbor expansion:** In each iteration, a node gathers its current neighbors and their neighbors to form a candidate list. The algorithm computes similarities between the node and all candidates. Because each node’s candidate list is independent, these computations can be assigned to separate GPU thread blocks and executed in parallel at a massive scale.

**3. Candidate list update:** If the algorithm finds candidates that are closer than the node’s current neighbors, it swaps out the more distant neighbors and updates the node’s kNN list. Over multiple iterations, this process produces a much higher-quality approximate kNN graph.

**4. Convergence check:** As iterations progress, fewer neighbor updates occur. Once the number of updated connections drops below a set threshold, the algorithm stops, indicating the graph has effectively stabilized.

Because neighbor expansion and similarity computation for different nodes are fully independent, CAGRA maps each node’s NN-Descent workload to a dedicated GPU thread block. This design enables massive parallelism and makes graph construction orders of magnitude faster than traditional CPU-based methods.


### Step 2: Pruning the Graph with 2-Hop Detours

After NN-Descent completes, the resulting graph is accurate but overly dense. NN-Descent intentionally keeps extra candidate neighbors, and the random initialization phase introduces many weak or irrelevant edges. As a result, each node often ends up with a degree two times—or even several times—higher than the target degree.

To produce a compact and efficient graph, CAGRA applies 2-hop detour pruning.

The idea is straightforward: if node _A_ can reach node _B_ indirectly through a shared neighbor _C_ (forming a path A → C → B), and the distance of this indirect path is comparable to the direct distance between _A_ and _B_, then the direct edge A → B is considered redundant and can be removed.

![](https://assets.zilliz.com/2_hop_detours_d15eae8702.png)

A key advantage of this pruning strategy is that each edge’s redundancy check depends only on local information—the distances between the two endpoints and their shared neighbors. Because every edge can be evaluated independently, the pruning step is highly parallelizable and fits naturally onto GPU batch execution. 

As a result, CAGRA can prune the graph efficiently on GPUs, reducing storage overhead by **40–50%** while preserving search accuracy and improving traversal speed during query execution.


## GPU_CAGRA in Milvus: What’s Different?

While GPUs offer major performance advantages for graph construction, production environments face a practical challenge: GPU resources are far more expensive and limited than CPUs. If both index building and query execution depend solely on GPUs, several operational issues quickly emerge:

- **Low resource utilization:** Query traffic is often irregular and bursty, leaving GPUs idle for long periods and wasting expensive compute capacity.

- **High deployment cost:** Assigning a GPU to every query-serving instance drives up hardware costs, even though most queries do not fully utilize GPU performance.

- **Limited scalability:** The number of GPUs available directly caps how many service replicas you can run, restricting your ability to scale with demand.

- **Reduced flexibility:** When both index building and querying depend on GPUs, the system becomes tied to GPU availability and cannot easily shift workloads to CPUs.

To address these constraints, Milvus 2.6.1 introduces a flexible deployment mode for the GPU_CAGRA index through the `adapt_for_cpu` parameter. This mode enables a hybrid workflow: CAGRA uses the GPU to build a high-quality graph index, while query execution runs on the CPU—typically using HNSW as the search algorithm. 

In this setup, GPUs are used where they deliver the most value—fast, high-accuracy index construction—while CPUs handle large-scale query workloads in a far more cost-effective and scalable manner.

As a result, this hybrid approach is particularly well suited for workloads where:

- **Data updates are infrequent**, so index rebuilds are rare

- **Query volume is high**, requiring many inexpensive replicas

- **Cost sensitivity is high**, and GPU usage must be tightly controlled


### Understanding `adapt_for_cpu`

In Milvus, the `adapt_for_cpu` parameter controls how a CAGRA index is serialized to disk during index building and how it is deserialized into memory at load time. By changing this setting at build time and load time, Milvus can flexibly switch between GPU-based index construction and CPU-based query execution.

Different combinations of `adapt_for_cpu` at build time and load time result in four execution modes, each designed for a specific operational scenario.



| **Build Time (`adapt_for_cpu`)** | **Load Time (`adapt_for_cpu`)** |                                  **Execution Logic**                                 |                              **Recommended Scenario**                              |
| :------------------------------: | :-----------------------------: | :----------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------: |
|             **true**             |             **true**            |  Build with GPU_CAGRA → serialize as HNSW → deserialize as HNSW → **CPU querying**  |                 Cost-sensitive workloads; large-scale query serving                |
|             **true**             |            **false**            |  Build with GPU_CAGRA → serialize as HNSW → deserialize as HNSW → **CPU querying**  |        Ensuing queries fall back to the CPU when parameter mismatches occur        |
|             **false**            |             **true**            |  Build with GPU_CAGRA → serialize as CAGRA → deserialize as HNSW → **CPU querying** | Keeping the original CAGRA index for storage while enabling a temporary CPU search |
|             **false**            |            **false**            | Build with GPU_CAGRA → serialize as CAGRA → deserialize as CAGRA → **GPU querying** |               Performance-critical workloads where cost is secondary               |

**Note:** The `adapt_for_cpu` mechanism supports only one-way conversion. A CAGRA index can be converted into HNSW because the CAGRA graph structure preserves all neighbor relationships that HNSW needs. However, an HNSW index cannot be converted back to CAGRA, as it lacks the additional structural information needed for GPU-based querying. As a result, the build-time settings should be selected carefully, with consideration for long-term deployment and querying requirements.


## Putting GPU_CAGRA to the Test

To evaluate the effectiveness of the hybrid execution model—using GPUs for index construction and CPUs for query execution—we conducted a series of controlled experiments in a standardized environment. The evaluation focuses on three dimensions: **index build performance**, **query performance**, and **recall accuracy**.

**Experimental Setup**

The experiments were performed on widely adopted, industry-standard hardware to ensure the results remain reliable and broadly applicable.

- CPU: MD EPYC 7R13 Processor(16 cpus)

- GPU: NVIDIA L4


### 1. Index Build Performance

We compare CAGRA built on the GPU with HNSW built on the CPU, under the same target graph degree of 64.

![](https://assets.zilliz.com/cp1_a177200ab2.png)

**Key Findings**

- **GPU CAGRA builds indexes 12–15× faster than CPU HNSW.** On both Cohere1M and Gist1M, GPU-based CAGRA significantly outperforms CPU-based HNSW, highlighting the efficiency of GPU parallelism during graph construction.

- **Build time increases linearly with NN-Descent iterations.** As iteration counts rise, build time grows in a near-linear manner, reflecting the iterative refinement nature of NN-Descent and providing a predictable trade-off between build cost and graph quality.


### 2. Query performance

In this experiment, the CAGRA graph is built once on the GPU and then queried using two different execution paths:

- **CPU querying**: the index is deserialized into HNSW format and searched on the CPU

- **GPU querying**: search runs directly on the CAGRA graph using GPU-based traversal

![](https://assets.zilliz.com/cp2_bd00e60553.png)

**Key Findings**

- **GPU search throughput is 5–6× higher than CPU search.** Across both Cohere1M and Gist1M, GPU-based traversal delivers substantially higher QPS, highlighting the efficiency of parallel graph navigation on GPUs.

- **Recall increases with NN-Descent iterations, then plateaus.** As the number of build iterations grows, recall improves for both CPU and GPU querying. However, beyond a certain point, additional iterations yield diminishing gains, indicating that graph quality has largely converged.


### 3. Recall accuracy

In this experiment, both CAGRA and HNSW are queried on the CPU to compare recall under identical query conditions.

![](https://assets.zilliz.com/cp3_1a46a7bdda.png)

**Key Findings**

**CAGRA achieves higher recall than HNSW on both datasets**, showing that even when a CAGRA index is built on the GPU and deserialized for CPU search, the graph quality is well preserved.


## What’s Next: Scaling Index Construction with Vamana

Milvus’s hybrid GPU–CPU approach offers a practical and cost-efficient solution for today’s large-scale vector search workloads. By building high-quality CAGRA graphs on GPUs and serving queries on CPUs, it combines fast index construction with scalable, affordable query execution—**particularly well suited for workloads with infrequent updates, high query volumes, and strict cost constraints.**

At even larger scales—**tens or hundreds of billions of vectors**—index construction itself becomes the bottleneck. When the full dataset no longer fits into GPU memory, the industry typically turns to **insert-based graph construction** methods such as **Vamana**. Instead of building the graph all at once, Vamana processes data in batches, incrementally inserting new vectors while maintaining global connectivity.

Its construction pipeline follows three key stages:

**1. Geometric batch growth** — beginning with small batches to form a skeleton graph, then increasing batch size to maximize parallelism, and finally using large batches to refine details.

**2. Greedy insertion** — each new node is inserted by navigating from a central entry point, iteratively refining its neighbor set.

**3. Backward edge updates** — adding reverse connections to preserve symmetry and ensure efficient graph navigation.

Pruning is integrated directly into the construction process using the α-RNG criterion: if a candidate neighbor _v_ is already covered by an existing neighbor _p′_ (i.e., _d(p′, v) < α × d(p, v)_), then _v_ is pruned. The parameter α allows precise control over sparsity and accuracy. GPU acceleration is achieved through in-batch parallelism and geometric batch scaling, striking a balance between index quality and throughput.

Together, these techniques enable teams to handle rapid data growth and large-scale index updates without running into GPU memory limitations.

![](https://assets.zilliz.com/One_more_thing_b458360e25.png)

The Milvus team is actively building Vamana support, with a targeted release in the first half of 2026. Stay tuned. 

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


## Learn More about Milvus 2.6 Features

- [Introducing Milvus 2.6: Affordable Vector Search at Billion Scale](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)

- [Introducing the Embedding Function: How Milvus 2.6 Streamlines Vectorization and Semantic Search](https://milvus.io/blog/data-in-and-data-out-in-milvus-2-6.md)

- [JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility](https://milvus.io/blog/json-shredding-in-milvus-faster-json-filtering-with-flexibility.md)

- [Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus](https://milvus.io/blog/unlocking-true-entity-level-retrieval-new-array-of-structs-and-max-sim-capabilities-in-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)



Optimizing CAGRA in Milvus: A Hybrid GPU–CPU Approach


Optimizing NVIDIA CAGRA in Milvus: A Hybrid GPU–CPU Approach to Faster Indexing and Cheaper Queries


AISAQ in Milvus Cuts Memory 3,200× for Billion-Scale Search


Introducing AISAQ in Milvus: Billion-Scale Vector Search Just Got 3,200× Cheaper on Memory



As AI systems are increasingly applied to real-time decision-making, geospatial data becomes increasingly important in a growing set of applications—particularly those that operate in the physical world or serve users across real locations. 

Consider food delivery platforms like DoorDash or Uber Eats. When a user places an order, the system isn’t simply calculating the shortest distance between two points. It evaluates restaurant quality, courier availability, live traffic conditions, service areas, and increasingly, user and item embeddings that represent personal preferences. Similarly, autonomous vehicles must perform path planning, obstacle detection, and scene-level semantic understanding under strict latency constraints—often within milliseconds. In these domains, effective decisions depend on combining spatial constraints with semantic similarity, rather than treating them as independent steps.

At the data layer, however, spatial and semantic data have traditionally been handled by separate systems.

- Geospatial databases and spatial extensions are designed to store coordinates, polygons, and spatial relationships such as containment or distance. 

- Vector databases handle vector embeddings that represent the data’s semantic meaning. 

When applications need both, they are often forced into multi-stage query pipelines—filtering by location in one system, then performing vector search in another. This separation increases system complexity, adds query latency, and makes it difficult to perform spatial–semantic reasoning efficiently at scale.

[Milvus 2.6](https://milvus.io/docs/release_notes.md#v264) addresses this problem by introducing the [Geometry Field](https://milvus.io/docs/geometry-field.md), which allows vector similarity search to be combined directly with spatial constraints. This enables use cases such as:

- Location-Base Service (LBS): “find similar POIs within this city block”

- Multi‑modal search: “retrieve similar photos within 1km of this point”

- Maps & logistics: “assets inside a region” or “routes intersecting a path”

Paired with the new [RTREE index](https://milvus.io/docs/rtree.md)—a tree-based structure optimized for spatial filtering—Milvus now supports efficient geospatial operators like `st_contains`, `st_within`, and `st_dwithin` alongside high-dimensional vector search. Together, they make spatially aware intelligent retrieval not just possible, but practical.

In this post, we’ll walk through how the Geometry Field and RTREE index work, and how they combine with vector similarity search to enable real-world, spatial-semantic applications.


## What Is a Geometry Field in Milvus?

A **Geometry field** is a schema-defined data type (`DataType.GEOMETRY`) in Milvus used to store geometric data. Unlike systems that handle only raw coordinates, Milvus supports a range of spatial structures—including **Point**, **LineString**, and **Polygon**. 

This makes it possible to represent real-world concepts such as restaurant locations (Point), delivery zones (Polygon), or autonomous-vehicle trajectories (LineString), all within the same database that stores semantic vectors. In other words, Milvus becomes a unified system for both _where_ something is and _what it means_.

Geometry values are stored using the [Well-Known Text (WKT)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) format, a human-readable standard for inserting and querying geometric data. This simplifies data ingestion and querying because WKT strings can be inserted directly into a Milvus record. For example:

```
data = [
    { 
        "id": 1,
        "geo": "POINT(116.4074 39.9042)",
        "vector": vector,
    }
]
```

## What Is the RTREE Index and How Does It Work?

Once Milvus introduces the Geometry data type, it also needs an efficient way to filter spatial objects. Milvus handles this using a two-stage spatial filtering pipeline:

- **Coarse filtering:** Quickly narrows down candidates using spatial indexes such as RTREE.

- **Fine filtering:** Applies exact geometry checks on the candidates that remain, ensuring correctness at boundaries.

This design balances performance and accuracy. The spatial index aggressively prunes irrelevant data, while precise geometric checks ensure correct results for operators such as containment, intersection, and distance thresholds.

At the core of this pipeline is **RTREE (Rectangle Tree)**, a spatial indexing structure designed to accelerate queries over geometric data. RTREE works by organizing objects hierarchically using **Minimum Bounding Rectangles (MBRs)**, allowing large portions of the search space to be skipped during query execution.

### Phase 1: Building the RTREE Index

RTREE construction follows a bottom-up process that groups nearby spatial objects into increasingly larger bounding regions:

**1. Create leaf nodes:** For each geometry object, calculate its **Minimum Bounding Rectangle (MBR)**—the smallest rectangle that fully contains the object—and store it as a leaf node.

**2. Group into larger boxs:** Cluster nearby leaf nodes and wrap each group inside a new MBR, producing internal nodes.

**3. Add the root node:** Create a root node whose MBR covers all internal groups, forming a height-balanced tree structure.

![](https://assets.zilliz.com/RTREE_Index_11b5d09e07.png)

**Phase 2: Accelerate queries**

**1. Form the query MBR:** Calculate the MBR for the geometry used in your query.

**2. Prune branches:** Starting from the root, compare the query MBR with each internal node. Skip any branch whose MBR does not intersect with the query MBR.

**3. Collect candidates:** Descend into intersecting branches and gather the candidate leaf nodes.

**4. Perform exact matching:** For each candidate, run the spatial predicate to get precise results.


### Why RTREE Is Fast

RTREE delivers strong performance in spatial filtering because of several key design features:

- **Every node stores an MBR:** Each node approximates the area of all geometries in its subtree. This makes it easy to decide whether a branch should be explored during a query.

- **Fast pruning:** Only subtrees whose MBR intersects the query region are explored. Irrelevant areas are ignored entirely.

- **Scales with data size:** RTREE supports spatial searches in **O(log N)** time, enabling fast queries even as the dataset expands.

- **Boost.Geometry implementation:** Milvus builds its RTREE index using [Boost.Geometry](https://www.boost.org/library/latest/geometry/), a widely used C++ library that provides optimized geometry algorithms and a thread-safe RTREE implementation suitable for concurrent workloads.


### Supported geometry operators

Milvus provides a set of spatial operators that allow you to filter and retrieve entities based on geometric relationships. These operators are essential for workloads that need to understand how objects relate to one another in space.

The following table lists the [geometry operators](https://milvus.io/docs/geometry-operators.md) currently available in Milvus.



|       **Operator**       |                                                  **Description**                                                 |
| :----------------------: | :--------------------------------------------------------------------------------------------------------------: |
| **st_intersects(A, B)** |                        Returns TRUE if geometries A and B share at least one common point.                       |
|  **st_contains(A, B)**  |                Returns TRUE if geometry A completely contains geometry B (excluding the boundary).               |
|   **st_within(A, B)**   | Returns TRUE if geometry A is completely contained within geometry B. This is the inverse of st_contains(A, B). |
|   **st_covers(A, B)**   |                      Returns TRUE if geometry A covers geometry B (including the boundary).                      |
|   **st_touches(A, B)**  |           Returns TRUE if geometries A and B touch at their boundaries but do not intersect internally.          |
|   **st_equals(A, B)**   |                            Returns TRUE if geometries A and B are spatially identical.                           |
|  **st_overlaps(A, B)**  |            Returns TRUE if geometries A and B partially overlap and neither fully contains the other.            |
| **st_dwithin(A, B, d)** |                          Returns TRUE if the distance between A and B is less than _d_.                          |


### How to Combine Geolocation Index and Vector Index

With Geometry support and the RTREE index, Milvus can combine geospatial filtering with vector similarity search in a single workflow. The process works in two steps:

**1. Filter by location using RTREE:** Milvus first uses the RTREE index to narrow the search to entities within the specified geographic range (e.g., “within 2 km”).

**2. Rank by semantics using vector search:** From the remaining candidates, the vector index selects the Top-N most similar results based on embedding similarity.

![](https://assets.zilliz.com/Geometry_R_Tree_f1d88fc252.png)

## Real-World Use Cases of Geo-Vector Retrieval

### 1. Delivery Services: Smarter, Location-Aware Recommendations

Platforms such as DoorDash or Uber Eats handle hundreds of millions of requests each day. The moment a user opens the app, the system must determine—based on the user’s location, time of day, taste preferences, estimated delivery times, real-time traffic, and courier availability—which restaurants or couriers are the best match _right now_.

Traditionally, this requires querying a geospatial database and a separate recommendation engine, followed by multiple rounds of filtering and re-ranking. With the Geolocation Index, Milvus greatly simplifies this workflow:

- **Unified storage** — Restaurant coordinates, courier locations, and user preference embeddings all live in one system.

- **Joint retrieval** — First apply a spatial filter (e.g., _restaurants within 3 km_), then use vector search to rank by similarity, taste preference, or quality.

- **Dynamic decision-making** — Combine real-time courier distribution and traffic signals to quickly assign the nearest, most suitable courier.

This unified approach allows the platform to perform spatial and semantic reasoning in a single query. For example, when a user searches “curry rice,” Milvus retrieves restaurants that are semantically relevant _and_ prioritizes those that are nearby, deliver quickly, and match the user’s historical taste profile.


### 2. Autonomous Driving: More Intelligent Decisions

In autonomous driving, geospatial indexing is fundamental to perception, localization, and decision-making. Vehicles must continuously align themselves to high-definition maps, detect obstacles, and plan safe trajectories—all within just a few milliseconds.

With Milvus, the Geometry type and RTREE index can store and query rich spatial structures such as:

- **Road boundaries** (LineString)

- **Traffic regulation zones** (Polygon)

- **Detected obstacles** (Point)

These structures can be indexed efficiently, allowing geospatial data to take part directly in the AI decision loop. For example, an autonomous vehicle can quickly determine whether its current coordinates fall within a specific lane or intersect with a restricted area, simply through an RTREE spatial predicate.

When combined with vector embeddings generated by the perception system—such as scene embeddings that capture the current driving environment—Milvus can support more advanced queries, like retrieving historical driving scenarios similar to the current one within a 50-meter radius. This helps models interpret the environment faster and make better decisions.


## Conclusion

Geolocation is more than latitude and longitude. In location-sensitive applications, it provides essential context about **where events occur, how entities relate spatially, and how those relationships shape system behavior**. When combined with semantic signals from machine learning models, geospatial data enables a richer class of queries that are difficult to express—or inefficient to execute—when spatial and vector data are handled separately.

With the introduction of the Geometry Field and the RTREE index, Milvus brings vector similarity search and spatial filtering into a single query engine. This allows applications to perform joint retrieval across **vectors, geospatial data, and time**, supporting use cases such as spatially aware recommendation systems, multimodal location-based search, and region- or path-constrained analytics. More importantly, it reduces architectural complexity by eliminating multi-stage pipelines that move data between specialized systems.

As AI systems continue to move closer to real-world decision-making, understanding **_what_** content is relevant will increasingly need to be paired with **_where_** it applies and **_when_** it matters. Milvus provides the building blocks for this class of spatial-semantic workloads in a way that is both expressive and practical for operating at scale.

For more information about the Geometry Field and the RTREE index, check the documentation below: 

- [Geometry Field | Milvus Documentation](https://milvus.io/docs/geometry-field.md)

- [RTREE | Milvus Documentation](https://milvus.io/docs/rtree.md)

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


## Learn More about Milvus 2.6 Features

- [Introducing Milvus 2.6: Affordable Vector Search at Billion Scale](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)

- [Introducing the Embedding Function: How Milvus 2.6 Streamlines Vectorization and Semantic Search](https://milvus.io/blog/data-in-and-data-out-in-milvus-2-6.md)

- [JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility](https://milvus.io/blog/json-shredding-in-milvus-faster-json-filtering-with-flexibility.md)

- [Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus](https://milvus.io/blog/unlocking-true-entity-level-retrieval-new-array-of-structs-and-max-sim-capabilities-in-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus ](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)

- [Vector Search in the Real World: How to Filter Efficiently Without Killing Recall](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md)

Geospatial Filtering + Vector Search in Milvus with Geometry Fields and RTREE


Bringing Geospatial Filtering and Vector Search Together with Geometry Fields and RTREE in Milvus 2.6


Array of Structs in Milvus: Entity-Level Multi-Vector Retrieval


Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus



Modern AI systems are producing more semi-structured JSON data than ever before. Customer and product information are compacted to a JSON object, microservices emit JSON logs on every request, IoT devices stream sensor readings in lightweight JSON payloads, and today’s AI applications increasingly standardize on JSON for structured output. The result is a flood of JSON-like data flowing into vector databases. 

Traditionally, there are two ways to handle JSON documents:

- **Predefine every field of JSON into a fixed schema and build an index:** This approach delivers solid query performance, but it’s rigid. Once the data format changes, every new or modified field triggers another round of painful Data Definition Language (DDL) updates and schema migrations.

- **Store the entire JSON object as a single column (both JSON type and Dynamic Schema in Milvus use this approach):** This option offers excellent flexibility, but at the cost of query performance. Each request requires runtime JSON parsing and often a full table scan, resulting in latency that spikes as the dataset grows.

It used to be a dilemma of flexibility and performance. 

Not anymore with the newly introduced JSON Shredding feature in [Milvus](https://milvus.io/). 

With the introduction of [JSON Shredding](https://milvus.io/docs/json-shredding.md), Milvus now achieves schema-free agility with the performance of columnar storage, finally making large-scale semi-structured data both flexible and query-friendly.

## How JSON Shredding Works

JSON shredding speeds up JSON queries by transforming row-based JSON documents into highly optimized columnar storage. Milvus preserves the flexibility of JSON for data modeling while automatically optimizing columnar storage—significantly improving data access and query performance.

To handle sparse or rare JSON fields efficiently, Milvus also has an inverted index for shared keys. All of this happens transparently to users: you can insert JSON documents as usual, and leave it to Milvus to manage the optimal storage and indexing strategy internally.

When Milvus receives raw JSON records with varying shapes and structures, it analyzes each JSON key for its occurrence ratio and type stability (whether its data type is consistent across documents). Based on this analysis, each key is classified into one of three categories:

- **Typed keys:** Keys that appear in most documents and always have the same data type (e.g., all integers or all strings).

- **Dynamic keys**: Keys that appear frequently but have mixed data types (e.g., sometimes a string, sometimes an integer).

- **Shared keys:** Keys that are infrequent, sparse, or nested, falling below a configurable frequency threshold.

Milvus handles each category differently to maximize efficiency:

- **Typed keys** are stored in dedicated, strongly typed columns.

- **Dynamic keys** are placed into dynamic columns based on the actual value type observed at runtime.

- Both typed and dynamic columns are stored in Arrow/Parquet columnar formats for fast scanning and highly optimized query execution.

- **Shared keys** are consolidated into a compact binary-JSON column, accompanied by a shared-key inverted index. This index accelerates queries on low-frequency fields by pruning irrelevant rows early and restricting the search to only those documents that contain the queried key.

This combination of adaptive columnar storage and inverted indexing forms the core of Milvus’s JSON shredding mechanism, enabling both flexibility and high performance at scale.

The overall workflow is illustrated below:

![](https://assets.zilliz.com/json_shredding_79a62a9661.PNG)

Now that we’ve covered the basics of how JSON Shredding works, let’s take a closer look at the key capabilities that make this approach both flexible and high-performance.

### Shredding and Columnarization

When a new JSON document is written, Milvus breaks it down and reorganizes it into optimized columnar storage:

- Typed and dynamic keys are automatically identified and stored in dedicated columns.

- If the JSON contains nested objects, Milvus generates path-based column names automatically. For example, a `name` field inside a `user` object can be stored with the column name `/user/name`.

- Shared keys are stored together in a single, compact binary JSON column. Because these keys appear infrequently, Milvus builds an inverted index for them, enabling fast filtering and allowing the system to quickly locate the rows that contain the specified key.

### Intelligent Column Management

Beyond shredding JSON into columns, Milvus adds an additional layer of intelligence through dynamic column management, ensuring that JSON Shredding stays flexible as data evolves.

- **Columns created as needed:** When new keys appear in incoming JSON documents, Milvus automatically groups values with the same key into a dedicated column. This preserves the performance advantages of columnar storage without requiring users to design schemas upfront. Milvus also infers the data type of new fields (e.g., INTEGER, DOUBLE, VARCHAR) and selects an efficient columnar format for them.

- **Every key is handled automatically:** Milvus analyzes and processes every key in the JSON document. This ensures broad query coverage without forcing users to predefine fields or build indexes in advance. 

### Query Optimization

Once the data is reorganized into the right columns, Milvus selects the most efficient execution path for each query:

- **Direct column scans for typed and dynamic keys:** If a query targets a field that has already been split into its own column, Milvus can scan that column directly. This reduces the total amount of data that needs to be processed and leverages SIMD-accelerated columnar computation for even faster execution.

- **Indexed lookup for shared keys:** If the query involves a field that was not promoted into its own column—typically a rare key—Milvus evaluates it against the shared-key column. The inverted index built on this column allows Milvus to quickly identify which rows contain the specified key and skip over the rest, significantly improving performance for low-frequency fields.

- **Automatic metadata management:** Milvus continuously maintains global metadata and dictionaries so that queries remain accurate and efficient, even as the structure of incoming JSON documents evolves over time.

## Performance benchmarks

We designed a benchmark to compare the query performance of storing the entire JSON document as a single raw field versus using the newly released JSON Shredding feature.

### Test environment and methodology

- Hardware: 1 core/8GB cluster

- Dataset: 1 million documents from [JSONBench](https://github.com/ClickHouse/JSONBench.git)

- Methodology: Measure QPS and latency across different query patterns

### Results: typed keys

This test measured performance when querying a key present in most documents.

| Query Expression          | QPS (without shredding) | QPS (with shredding) | Performance Boost |
| ------------------------- | ----------------------- | -------------------- | ----------------- |
| json['time_us'] > 0     | 8.69                    | 287.5                | **33x**           |
| json['kind'] == 'commit' | 8.42                    | 126.1                | **14.9x**         |

### Results: shared keys

This test focused on querying sparse, nested keys that fall into the “shared” category.

| Query Expression                     | QPS (without shredding) | QPS (with shredding) | Performance Boost |
| ------------------------------------ | ----------------------- | -------------------- | ----------------- |
| json['identity']['seq'] > 0        | 4.33                    | 385                  | **88.9x**         |
| json['identity']['did'] == 'xxxxx' | 7.6                     | 352                  | **46.3x**         |

Shared-key queries show the most dramatic improvements (up to 89× faster), while typed-key queries deliver consistent 15–30× speedups. Overall, every query type benefits from JSON Shredding, with clear performance gains across the board.

## Try It Now

Whether you’re working with API logs, IoT sensor data, or rapidly evolving application payloads, JSON Shredding gives you the rare ability to have both flexibility and high performance. 

The feature is now available and welcome to try it out now. You can also check [this doc](https://milvus.io/docs/json-shredding.md) for more details.

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Milvus JSON Shredding: Faster JSON Filtering With Flexibility


JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility




If you’ve ever built a vector search application, you already know the workflow a little too well. Before any data can be stored, it must first be transformed into vectors using an embedding model, cleaned and formatted, and then finally ingested into your vector database. Every query goes through the same process as well: embed the input, run a similarity search, then map the resulting IDs back to your original documents or records. It works — but it creates a distributed tangle of preprocessing scripts, embedding pipelines, and glue code that you have to maintain.

[Milvus](https://milvus.io/), a high-performance open-source vector database, now takes a major step toward simplifying all of that. [Milvus 2.6](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md) introduces the **Data-in, Data-out feature (also known as the** [**Embedding Function**](https://milvus.io/docs/embedding-function-overview.md#Embedding-Function-Overview)**)**, a built-in embedding capability that connects directly to major model providers such as OpenAI, AWS Bedrock, Google Vertex AI, and Hugging Face. Instead of managing your own embedding infrastructure, Milvus can now call these models for you. You can also insert and query using raw text — and soon other data types — while Milvus automatically handles vectorization at write and query time. 

In the rest of this post, we’ll take a closer look at how Data-in, Data-out works under the hood, how to configure providers and embedding functions, and how you can use it to streamline your vector search workflows end-to-end. 


## What is Data-in, Data-out?

Data-in, Data-out in Milvus 2.6 is built on the new Function module — a framework that enables Milvus to handle data transformation and embedding generation internally, without any external preprocessing services. (You can follow the design proposal in [GitHub issue #35856](https://github.com/milvus-io/milvus/issues/35856).) With this module, Milvus can take raw input data, call an embedding provider directly, and automatically write the resulting vectors into your collection.

At a high level, the **Function** module turns embedding generation into a native database capability. Instead of running separate embedding pipelines, background workers, or reranker services, Milvus now sends requests to your configured provider, retrieves embeddings, and stores them alongside your data — all inside the ingestion path. This removes the operational overhead of managing your own embedding infrastructure.

Data-in, Data-out introduces three major improvements to the Milvus workflow:

- **Insert raw data directly** – You can now insert unprocessed text, images, or other data types directly into Milvus. No need to convert them into vectors in advance.

- **Configure one embedding function** – Once you configure an embedding model in Milvus, it automatically manages the entire embedding process. Milvus integrates seamlessly with a range of model providers, including OpenAI, AWS Bedrock, Google Vertex AI, Cohere, and Hugging Face.

- **Query with raw inputs** – You can now perform semantic search using raw text or other content-based queries. Milvus uses the same configured model to generate embeddings on the fly, perform similarity search, and return relevant results.

In short, Milvus now automatically embeds — and optionally reranks — your data. Vectorization becomes a built-in database function, eliminating the need for external embedding services or custom preprocessing logic.


## How Data-in, Data-out Works

The diagram below illustrates how Data-in, Data-out operates inside Milvus. 

![](https://assets.zilliz.com/diagram_data_in_data_out_4c9e06c884.png)

The Data-in, Data-out workflow can be broken down into six main steps:

1. **Input Data** – The user inserts raw data — such as text, images, or other content types — directly into Milvus without performing any external preprocessing.

2. **Generate Embeddings** – The Function module automatically invokes the configured embedding model through its third-party API, converting the raw input into vector embeddings in real time.

3. **Store Embeddings** – Milvus writes the generated embeddings into the designated vector field within your collection, where they become available for similarity search operations.

4. **Submit a Query** – The user issues a raw-text or content-based query to Milvus, just as with the input stage.

5. **Semantic Search** – Milvus embeds the query using the same configured model, runs a similarity search over the stored vectors, and determines the closest semantic matches.

6. **Return Results** – Milvus returns the top-k most similar results — mapped back to their original data — directly to the application.


## How to Configure Data-in, Data-out

### Prerequisites

- Install the latest version of **Milvus 2.6**.

- Prepare your embedding API key from a supported provider (e.g., OpenAI, AWS Bedrock, or Cohere). In this example, we’ll use **Cohere** as the embedding provider.


### Modify the `milvus.yaml` Configuration

If you are running Milvus with **Docker Compose**, you’ll need to modify the `milvus.yaml` file to enable the Function module. You can refer to the official documentation for guidance: [Configure Milvus with Docker Compose](https://milvus.io/docs/configure-docker.md?tab=component#Download-a-configuration-file) (Instructions for other deployment methods can also be found here).

In the configuration file, locate the sections `credential` and `function`.

Then, update the fields `apikey1.apikey` and `providers.cohere`.

```
...
credential:
  aksk1:
    access_key_id:  # Your access_key_id
    secret_access_key:  # Your secret_access_key
  apikey1:
    apikey: "***********************" # Edit this section
  gcp1:
    credential_json:  # base64 based gcp credential data
# Any configuration related to functions
function:
  textEmbedding:
    providers:
                        ...
      cohere: # Edit the section below
        credential:  apikey1 # The name in the crendential configuration item
        enable: true # Whether to enable cohere model service
        url:  "https://api.cohere.com/v2/embed" # Your cohere embedding url, Default is the official embedding url
      ...
...
```



Once you’ve made these changes, restart Milvus to apply the updated configuration.


## How to Use the Data-in, Data-out Feature

### 1. Define the Schema for the Collection 

To enable the embedding feature, your **collection schema** must include at least three fields:

- **Primary key field (**`id`**)** – Uniquely identifies each entity in the collection.

- **Scalar field (**`document`**)** – Stores the original raw data.

- **Vector field (**`dense`**)** – Stores the generated vector embeddings.

```
from pymilvus import MilvusClient, DataType, Function, FunctionType
# Initialize Milvus client
client = MilvusClient(
    uri="http://localhost:19530",
)
# Create a new schema for the collection
schema = client.create_schema()
# Add primary field "id"
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=False)
# Add scalar field "document" for storing textual data
schema.add_field("document", DataType.VARCHAR, max_length=9000)
# Add vector field "dense" for storing embeddings.
# IMPORTANT: Set `dim` to match the exact output dimension of the embedding model.
# For instance, OpenAI's text-embedding-3-small model outputs 1536-dimensional vectors.
# For dense vector, data type can be FLOAT_VECTOR or INT8_VECTOR
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=1536) # Set dim according to the embedding model you use.
```


### 2. Define the Embedding Function

Next, define the **embedding function** in the schema. 

- `name` – A unique identifier for the function.

- `function_type` – Set to `FunctionType.TEXTEMBEDDING` for text embeddings. Milvus also supports other function types such as `FunctionType.BM25` and `FunctionType.RERANK`. See [Full Text Search](https://milvus.io/docs/full-text-search.md#Full-Text-Search) and [Decay Ranker Overview](https://milvus.io/docs/decay-ranker-overview.md#Decay-Ranker-Overview) for more details.

- `input_field_names` – Defines the input field for raw data (`document`).

- `output_field_names` – Defines the output field where the vector embeddings will be stored (`dense`).

- `params` – Contains configuration parameters for the embedding function. The values for `provider` and `model_name` must match the corresponding entries in your `milvus.yaml` configuration file.

**Note:** Each function must have a unique `name` and `output_field_names` to distinguish different transformation logics and prevent conflicts.



```
# Define embedding function (example: OpenAI provider)
text_embedding_function = Function(
    name="cohere_embedding",                  # Unique identifier for this embedding function
    function_type=FunctionType.TEXTEMBEDDING, # Type of embedding function
    input_field_names=["document"],           # Scalar field to embed
    output_field_names=["dense"],             # Vector field to store embeddings
    params={                                  # Provider-specific configuration (highest priority)
        "provider": "cohere",                 # Embedding model provider
        "model_name": "embed-v4.0",     # Embedding model
        # "credential": "apikey1",            # Optional: Credential label
        # Optional parameters:
        # "dim": "1536",       # Optionally shorten the vector dimension
        # "user": "user123"    # Optional: identifier for API tracking
    }
)
# Add the embedding function to your schema
schema.add_function(text_embedding_function)
```




### 3. Configure the Index

Once the fields and functions are defined, create an index for the collection. For simplicity, we use the AUTOINDEX type here as an example.



```
# Prepare index parameters
index_params = client.prepare_index_params()
# Add AUTOINDEX to automatically select optimal indexing method
index_params.add_index(
    field_name="dense",
    index_type="AUTOINDEX",
    metric_type="COSINE" 
)
```




### 4. Create the Collection

Use the defined schema and index to create a new collection. In this example, we’ll create a collection named Demo.



```
# Create collection named "demo"
client.create_collection(
    collection_name='demo', 
    schema=schema, 
    index_params=index_params
)
```




### 5. Insert Data

Now you can insert raw data directly into Milvus — there’s no need to generate embeddings manually. 



```
# Insert sample documents
client.insert('demo', [
    {'id': 1, 'document': 'Milvus simplifies semantic search through embeddings.'},
    {'id': 2, 'document': 'Vector embeddings convert text into searchable numeric data.'},
    {'id': 3, 'document': 'Semantic search helps users find relevant information quickly.'},
])
```




### 6. Perform Vector Search

After inserting data, you can perform searches directly using raw text queries. Milvus automatically converts your query into an embedding, perform similarity search against stored vectors, and return the top matches.



```
# Perform semantic search
results = client.search(
    collection_name='demo', 
    data=['How does Milvus handle semantic search?'], # Use text query rather than query vector
    anns_field='dense',   # Use the vector field that stores embeddings
    limit=1,
    output_fields=['document'],
)
print(results)
# Example output:
# data: ["[{'id': 1, 'distance': 0.8821347951889038, 'entity': {'document': 'Milvus simplifies semantic search through embeddings.'}}]"]
```



For more details on vector search, see: [Basic Vector Search ](https://milvus.io/docs/single-vector-search.md)and [Query API](https://milvus.io/docs/get-and-scalar-query.md).


## Get Started with Milvus 2.6 

With Data-in, Data-out, Milvus 2.6 takes vector search simplicity to the next level. By integrating embedding and reranking functions directly within Milvus, you no longer need to manage external preprocessing or maintain separate embedding services.

Ready to try it out? Install [Milvus](https://milvus.io/docs) 2.6 today and experience the power of Data-in, Data-out for yourself. 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


## Learn More about Milvus 2.6 Features

- [Introducing Milvus 2.6: Affordable Vector Search at Billion Scale](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)

- [JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility](https://milvus.io/blog/json-shredding-in-milvus-faster-json-filtering-with-flexibility.md)

- [Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus](https://milvus.io/blog/unlocking-true-entity-level-retrieval-new-array-of-structs-and-max-sim-capabilities-in-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus ](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)

- [Vector Search in the Real World: How to Filter Efficiently Without Killing Recall ](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md)


Introducing the Embedding Function: How Milvus 2.6 Streamlines Vectorization and Semantic Search


7 Years, 2 Major Rebuilds, 40K+ GitHub Stars: The Rise of Milvus as the Leading Open-Source Vector Database


Choosing a vector database for ANN search at Reddit


RAG-Anything and Milvus for Multimodal RAG Systems

Multimodal RAG Made Simple: RAG-Anything + Milvus Instead of 20 Separate Tools



Most developers start their agentic systems with a single agent and only later realize they’ve basically built a very expensive chatbot. For simple tasks, a ReAct-style agent works fine, but it quickly hits limits: it can’t run steps in parallel, it loses track of long reasoning chains, and it tends to fall apart once you add too many tools to the mix. Multi-agent setups promise to fix this, but they bring their own problems: coordination overhead, brittle handoffs, and a ballooning shared context that quietly erodes model quality.

[OpenAgents](https://github.com/OpenAgentsInc) is an open-source framework for building multi-agent systems in which AI agents work together, share resources, and tackle long-horizon projects within persistent communities. Instead of a single central orchestrator, OpenAgents lets agents collaborate in a more distributed way: they can discover each other, communicate, and coordinate around shared goals.

Paired with the [Milvus](https://milvus.io/) vector database, this pipeline gains a scalable, high-performance long-term memory layer. Milvus powers agent memory with fast semantic search, flexible indexing choices like HNSW and IVF, and clean isolation through partitioning, so agents can store, retrieve, and reuse knowledge without drowning in context or stepping on each other’s data.

In this post, we’ll walk through how OpenAgents enables distributed multi-agent collaboration, why Milvus is a critical foundation for scalable agent memory, and how to assemble such a system step by step.


## Challenges in Building Real-World Agent Systems

Many mainstream agent frameworks today—LangChain, AutoGen, CrewAI, and others—are built around a **task-centric** model. You spin up a set of agents, give them a job, maybe define a workflow, and let them run. This works well for narrow or short-lived use cases, but in real production environments, it exposes three structural limitations:

- **Knowledge remains siloed.** An agent’s experience is confined to its own deployment. A code-review agent in engineering doesn’t share what it learns with a product-team agent evaluating feasibility. Every team ends up rebuilding knowledge from scratch, which is both inefficient and brittle. 

- **Collaboration is rigid.** Even in multi-agent frameworks, cooperation usually depends on workflows defined in advance. When collaboration needs to shift, these static rules cannot adapt, making the entire system less flexible.

- **A lack of a persistent state.** Most agents follow a simple lifecycle: _start → execute → shut down._ They forget everything between runs—context, relationships, decisions made, and interaction history. Without a persistent state, agents cannot build long-term memory or evolve their behavior.

These structural issues come from treating agents as isolated task executors rather than participants in a broader collaborative network. 

The OpenAgents team believes that future agent systems need more than stronger reasoning—they need a mechanism that enables agents to discover one another, build relationships, share knowledge, and work together dynamically. And critically, this should not depend on a single central controller. The internet works because it’s distributed—no single node dictates everything, and the system becomes more robust and scalable as it grows. Multi-agent systems benefit from the same design principle. That’s why OpenAgents removes the idea of an all-powerful orchestrator and instead enables decentralized, network-driven cooperation.


## What’s OpenAgents? 

OpenAgents is an open-source framework for building AI agent networks that enables open collaboration, where AI agents work together, share resources, and tackle long-horizon projects. It provides the infrastructure for an internet of agents — where agents collaborate openly with millions of other agents in persistent, growing communities. At the technical level, the system is structured around three core components: **Agent Network, Network Mods, and Transports.**


### 1. Agent Network: A Shared Environment for Collaboration

An agent network is a shared environment where multiple agents can connect, communicate, and work together to solve complex tasks. Its core characteristics include:

- **Persistent operation:** Once created, the Network stays online independently of any single task or workflow.

- **Dynamic agent:** Agents can join at any time using a Network ID; no pre-registration required.

- **Multi-protocol support:** A unified abstraction layer supports communication over WebSocket, gRPC, HTTP, and libp2p.

- **Autonomous configuration:** Each Network maintains its own permissions, governance, and resources.

With just one line of code, you can spin up a Network, and any agent can join immediately through standard interfaces.


### 2. Network Mods: Pluggable Extensions for Collaboration

Mods provide a modular layer of collaboration features that stay decoupled from the core system. You can mix and match Mods based on your specific needs, enabling collaboration patterns tailored to each use case.


| **Mod**                 | **Purpose**                     | **Use cases**                                   |
| ----------------------- | ------------------------------- | ----------------------------------------------- |
| **Workspace Messaging** | Real-time message communication | Streaming responses, instant feedback           |
| **Forum**               | Asynchronous discussion         | Proposal reviews, multi-round deliberation      |
| **Wiki**                | Shared knowledge base           | Knowledge consolidation, document collaboration |
| **Social**              | Relationship graph              | Expert routing, trust networks                  |

All Mods operate on a unified event system, making it easy to extend the framework or introduce custom behaviors whenever required.


### 3. Transports: A Protocol-Agnostic Channel for Communication

Transports are the communication protocols that allow heterogeneous agents to connect and exchange messages within an OpenAgents network. OpenAgents supports multiple transport protocols that can run simultaneously inside the same network, including:

- **HTTP/REST** for broad, cross-language integration

- **WebSocket** for low-latency, bidirectional communication

- **gRPC** for high-performance RPC suited to large-scale clusters

- **libp2p** for decentralized, peer-to-peer networking

- **A2A**, an emerging protocol designed specifically for agent-to-agent communication

All transports operate through a unified event-based message format, enabling seamless translation between protocols. You don’t need to worry about which protocol a peer agent uses—the framework handles it automatically. Agents built in any language or framework can join an OpenAgents network without rewriting existing code.

## Integrating OpenAgents with Milvus for Long-Term Agentic Memory

OpenAgents solves the challenge of how agents **communicate, discover each other, and collaborate**—but collaboration alone isn’t enough. Agents generate insights, decisions, conversation history, tool results, and domain-specific knowledge. Without a persistent memory layer, all of that evaporates the moment an agent shuts down.

This is where **Milvus** becomes essential. Milvus provides the high-performance vector storage and semantic retrieval needed to turn agent interactions into durable, reusable memory. When integrated into the OpenAgents network, it offers three major advantages:


#### **1. Semantic Search**

Milvus delivers fast semantic search using indexing algorithms like HNSW and IVF_FLAT. Agents can retrieve the most relevant historical records based on meaning rather than keywords, enabling them to:

- recall prior decisions or plans,

- avoid repeating work,

- maintain long-horizon context across sessions.

This is the backbone of _agentic memory_: fast, relevant, contextual retrieval.


#### **2. Billion-Scale Horizontal Scalability**

Real agent networks generate massive amounts of data. Milvus is built to operate comfortably at this scale, offering:

- storage and search over billions of vectors,

- < 30 ms latency even under high-throughput Top-K retrieval,

- a fully distributed architecture that scales linearly as demand grows.

Whether you have a dozen agents or thousands working in parallel, Milvus keeps retrieval fast and consistent.


#### **3. Multi-Tenant Isolation**

Milvus provides granular multi-tenant isolation through **Partition Key**, a lightweight partitioning mechanism that segments memory inside a single collection. This allows:

- different teams, projects, or agent communities to maintain independent memory spaces,

- dramatically lower overhead compared to maintaining multiple collections,

- optional cross-partition retrieval when shared knowledge is needed.

This isolation is crucial for large multi-agent deployments where data boundaries must be respected without compromising retrieval speed.

OpenAgents connects to Milvus through **custom Mods** that call Milvus APIs directly. Agent messages, tool outputs, and interaction logs are automatically embedded into vectors and stored in Milvus. Developers can customize:

- the embedding model,

- storage schema and metadata,

- and retrieval strategies (e.g., hybrid search, partitioned search).

This gives each agent community a memory layer that is scalable, persistent, and optimized for semantic reasoning.

## How to Build a Multi-Agent Chatbot with OpenAgent and Milvus

To make things concrete, let’s walk through a demo: building a **developer-support community** where multiple specialist agents—Python experts, database experts, DevOps engineers, and more—collaborate to answer technical questions. Instead of relying on a single overworked generalist agent, each expert contributes domain-specific reasoning, and the system routes queries to the best-suited agent automatically.

This example demonstrates how to integrate **Milvus** into an OpenAgents deployment to provide long-term memory for technical Q&A. Agent conversations, past solutions, troubleshooting logs, and user queries are all converted into vector embeddings and stored in Milvus, giving the network the ability to:

- remember previous answers,

- reuse prior technical explanations,

- maintain consistency across sessions, and

- improve over time as more interactions accumulate.


### Prerequisite

- python3.11+

- conda

- Openai-key


### 1. Define Dependencies

Define the Python packages required for the project:

```
# Core framework
openagents>=0.6.11
# Vector database
pymilvus>=2.5.1
# Embedding model
sentence-transformers>=2.2.0
# LLM integration
openai>=1.0.0
# Environment config
python-dotenv>=1.0.0
```

### 2. Environment Variables

Here is the template for your environment configuration:

```
# LLM configuration (required)
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-4o
# Milvus configuration
MILVUS_URI=./multi_agent_memory.db
# Embedding model configuration
EMBEDDING_MODEL=text-embedding-3-large
EMBEDDING_DIMENSION=3072
# Network configuration
NETWORK_HOST=localhost
NETWORK_PORT=8700
STUDIO_PORT=8050
```

### 3. Configure Your OpenAgents Network

Define the structure of your agent network and its communication settings:

```
# Network transport protocol (HTTP on port 8700)
# Multi-channel messaging system (general, coordination, expert channels)
# Agent role definitions (coordinator, python_expert, etc.)
# Milvus integration settings
network:
  name: "Multi-Agent Collaboration Demo"
  transports:
    - type: "http"
      config:
        port: 8700
        host: "localhost"
  mods:
    - name: "openagents.mods.workspace.messaging"
      config:
        channels:
          - name: "general"          # User question channel
          - name: "coordination"     # Coordinator channel
          - name: "python_channel"   # Python expert channel
          - name: "milvus_channel"   # Milvus expert channel
          - name: "devops_channel"   # DevOps expert channel
  agents:
    coordinator:
      type: "coordinator"
      description: "Coordinator Agent, responsible for analyzing queries and dispatching tasks to expert agents"
      channels: ["general", "coordination"]
    python_expert:
      type: "expert"
      domain: "python"
```

### 4. Implement Multi-Agent Collaboration

The following shows core code snippets (not the full implementation).

```
# SharedMemory: Milvus’s SharedMemory system
# CoordinatorAgent: Coordinator Agent, responsible for analyzing queries and dispatching tasks to expert agents
# PythonExpertAgent: Python Expert
# MilvusExpertAgent: Milvus Expert
# DevOpsExpertAgent: DevOps Expert
import os
import asyncio
import json
from typing import List, Dict
from dotenv import load_dotenv
from openagents.agents.worker_agent import WorkerAgent
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
import openai
load_dotenv()
class SharedMemory:
    """SharedMemory in Milvus for all Agents"""
    def __init__(self):
        connections.connect(uri="./multi_agent_memory.db")
        self.setup_collections()
        self.openai_client = openai.OpenAI(
            api_key=os.getenv("OPENAI_API_KEY"),
            base_url=os.getenv("OPENAI_BASE_URL")
        )
    def setup_collections(self):
        """Create memory collections: expert knowledge, collaboration history, problem solutions"""
        collections = {
            "expert_knowledge": "expert knowledge",
            "collaboration_history": "collaboration history", 
            "problem_solutions": "problem solutions"
        }
        # Code to create vector collections...
    async def search_knowledge(self, query: str, collection_name: str):
        """Search for relevant knowledge"""
        # Vector search implementation...
    async def store_knowledge(self, agent_id: str, content: str, metadata: dict, collection_name: str):
        """Store knowledge"""
        # Store into the vector database...
class CoordinatorAgent(WorkerAgent):
    """Coordinator Agent - analyzes questions and coordinates other Agent"""
    def __init__(self):
        super().__init__(agent_id="coordinator")
        self.expert_agents = {
            "python": "python_expert",
            "milvus": "milvus_expert", 
            "devops": "devops_expert"
        }
    async def analyze_question(self, question: str) -> List[str]:
        """Determine which experts are needed for the question"""
        keywords = {
            "python": ["python", "django", "flask", "async"],
            "milvus": ["milvus", "vector", "index", "performance"],
            "devops": ["deployment", "docker", "kubernetes", "operations"]
        }
        # Keyword matching logic...
        return needed_experts
    async def coordinate_experts(self, question: str, needed_experts: List[str]):
        """Coordinate collaboration among expert Agent"""
        # 1. Notify experts to begin collaborating
        # 2. Dispatch tasks to each expert
        # 3. Collect expert responses
        # 4. Return expert opinions
    async def on_channel_post(self, context):
        """Main logic for handling user questions"""
        content = context.incoming_event.payload.get('content', {}).get('text', '')
        if content and not content.startswith('🎯'):
            # 1. Analyze question → 2. Coordinate experts → 3. Merge answers → 4. Reply to user
class PythonExpertAgent(WorkerAgent):
    """Python Expert Agent"""
    async def analyze_python_question(self, question: str) -> str:
        """Analyze Python-related questions and provide expert advice"""
        # 1. Search for relevant experience
        # 2. Use LLM to generate expert response
        # 3. Store result in collaboration history
        return answer
# Start all Agens
async def run_multi_agent_demo():
    coordinator = CoordinatorAgent()
    python_expert = PythonExpertAgent()
    milvus_expert = MilvusExpertAgent()
    devops_expert = DevOpsExpertAgent()
    # Connect to the OpenAgents network
    await coordinator.async_start(network_host="localhost", network_port=8700)
    # ... Start other Agent
    while True:
        await asyncio.sleep(1)
if __name__ == "__main__":
    asyncio.run(run_multi_agent_demo())
```

### 5. Create and Activate a Virtual Environment

```
conda create -n openagents
conda activate openagents
```

**Install Dependencies**

```
pip install -r requirements.txt
```

**Configure API Keys**

```
cp .env.example .env
```

**Start the OpenAgents Network**

```
openagents network start .
```

![](https://assets.zilliz.com/network_169812ab94.png)

**Start the Multi-Agent Service**

```
python multi_agent_demo.py
```

![](https://assets.zilliz.com/multiagent_service_1661d4b91b.png)

**Start OpenAgents Studio**

```
openagents studio -s
```

![](https://assets.zilliz.com/start_studio_4cd126fea2.png)

**Access Studio**

```
http://localhost:8050
```

![](https://assets.zilliz.com/access_studio1_a33709914b.png)

![](https://assets.zilliz.com/access_studio3_293604c79e.png)

![](https://assets.zilliz.com/access_studio_3_8d98a4cfe8.png)

**Check the status of your agents and network:**

![](https://assets.zilliz.com/check_state_bba1a4fe16.png)

## Conclusion

OpenAgents provides the coordination layer that lets agents discover each other, communicate, and collaborate, while Milvus solves the equally critical problem of how knowledge is stored, shared, and reused. By delivering a high-performance vector memory layer, Milvus enables agents to build persistent context, recall past interactions, and accumulate expertise over time. Together, they push AI systems beyond the limits of isolated models and toward the deeper collaborative potential of a true multi-agent network. 

Of course, no multi-agent architecture is without trade-offs. Running agents in parallel can increase token consumption, errors may cascade across agents, and simultaneous decision-making can lead to occasional conflicts. These are active areas of research and ongoing improvement—but they don’t diminish the value of building systems that can coordinate, remember, and evolve.

🚀 Ready to give your agents long-term memory?

Explore [Milvus](https://milvus.io/) and try integrating it with your own workflow. 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Build Smarter Multi-Agent Systems with OpenAgents and Milvus

OpenAgents x Milvus: How to Build Smarter Multi-Agent Systems That Share Memory


Gemini 3 Pro + Milvus: Robust RAG With Advanced Reasoning and Multimodal Power


Gemini 3 Pro + Milvus: Building a More Robust RAG With Advanced Reasoning and Multimodal Power



Over the past few weeks, a surprisingly heated argument has erupted across X and Hacker News: _Do we actually need MCP servers anymore?_ Some developers claim MCP is over-engineered, token-hungry, and fundamentally misaligned with how agents should use tools. Others defend MCP as the reliable way to expose real-world capabilities to language models. Depending on which thread you read, MCP is either the future of tool use—or dead on arrival.

![](https://assets.zilliz.com/hackernews_c3236cca2c.png)

The frustration is understandable. MCP gives you robust access to external systems, but it also forces the model to load long schemas, verbose descriptions, and sprawling tool lists. That adds real cost. If you download a meeting transcript and later feed it into another tool, the model may reprocess the same text multiple times, inflating token usage for no obvious benefit. For teams operating at scale, this isn’t an inconvenience—it’s a bill.

But declaring MCP obsolete is premature. Anthropic—the same team that invented MCP—quietly introduced something new: [**Skills**](https://claude.com/blog/skills). Skills are lightweight Markdown/YAML definitions that describe _how_ and _when_ a tool should be used. Instead of dumping full schemas into the context window, the model first reads compact metadata and uses that to plan. In practice, Skills dramatically reduce token overhead and give developers more control over tool orchestration.

So, does this mean Skills will replace MCP? Not quite. Skills streamline planning, but MCP still provides the actual capabilities: reading files, calling APIs, interacting with storage systems, or plugging into external infrastructure like [**Milvus**](https://milvus.io/), an open-source vector database that underpins fast semantic retrieval at scale, making it a critical backend when your Skills need real data access.

This post breaks down what Skills do well, where MCP still matters, and how both fit into Anthropic’s evolving agent architecture. Then we’ll walk through how to build your own Skills that integrate cleanly with Milvus.

## What Are Anthropic Agent Skills and How Do They Work

A long-standing pain point of traditional AI agents is that instructions get washed out as the conversation grows.

Even with the most carefully crafted system prompts, the model’s behavior can gradually drift over the course of the conversation. After several turns, Claude begins to forget or lose focus on the original instructions.

The problem lies in the structure of the system prompt. It is a one-time, static injection that competes for space in the model’s context window, alongside conversation history, documents, and any other inputs. As the context window fills, the model’s attention to the system prompt becomes increasingly diluted, leading to a loss of consistency over time. 

Skills were designed to address this issue. Skills are folders containing instructions, scripts, and resources. Rather than relying on a static system prompt, Skills break down expertise into modular, reusable, and persistent instruction bundles that Claude can discover and load dynamically when needed for a task.

When Claude begins a task, it first performs a lightweight scan of all available Skills by reading only their YAML metadata (just a few dozen tokens). This metadata provides just enough information for Claude to determine if a Skill is relevant to the current task. If so, Claude expands into the full set of instructions (usually under 5k tokens), and additional resources or scripts are loaded only if necessary. 

This progressive disclosure allows Claude to initialize a Skill with just 30–50 tokens, significantly improving efficiency and reducing unnecessary context overhead.

![](https://assets.zilliz.com/how_skills_works_a8563f346c.png)

## How Skills Compares to Prompts, Projects, MCP, and Subagents

Today’s model tooling landscape can feel crowded. Even within Claude's agentic ecosystem alone, there are several distinct components: Skills, prompts, Projects, subagents, and MCP.

Now that we understand what Skills are and how they work through modular instruction bundles and dynamic loading, we need to know how Skills relate to other parts of the Claude ecosystem, especially MCP. Here is a summary:

### 1. Skills

Skills are folders that contain instructions, scripts, and resources. Claude discovers and loads them dynamically using progressive disclosure: first metadata, then full instructions, and finally any required files.

**Best for:**

- Organizational workflows (brand guidelines, compliance procedures)

- Domain expertise (Excel formulas, data analysis)

- Personal preferences (note-taking systems, coding patterns)

- Professional tasks that need to be reused across conversations (OWASP-based code security reviews)

### 2. Prompts

Prompts are the natural-language instructions you give Claude within a conversation. They are temporary and exist only in the current conversation.

**Best for:**

- One-off requests (summarizing an article, formatting a list)

- Conversational refinement (adjusting tone, adding details)

- Immediate context (analyzing specific data, interpreting content)

- Ad-hoc instructions

### 3. Projects

Projects are self-contained workspaces with their own chat histories and knowledge bases. Each project offers a 200K context window. When your project knowledge approaches context limits, Claude transitions seamlessly into RAG mode, allowing up to a 10x expansion in effective capacity.

**Best for:**

- Persistent context (e.g., all conversations related to a product launch)

- Workspace organization (separate contexts for different initiatives)

- Team collaboration (on Team and Enterprise plans)

- Custom instructions (project-specific tone or perspective)

### 4. Subagents

Subagents are specialized AI assistants with their own context windows, custom system prompts, and specific tool permissions. They can work independently and return results to the main agent.

**Best for:**

- Task specialization (code review, test generation, security audits)

- Context management (keep the main conversation focused)

- Parallel processing (multiple subagents working on different aspects simultaneously)

- Tool restriction (e.g., read-only access)

### 5. MCP (Model Context Protocol)

The Model Context Protocol (MCP) is an open standard that connects AI models to external tools and data sources.

**Best for:**

- Accessing external data (Google Drive, Slack, GitHub, databases)

- Using business tools (CRM systems, project management platforms)

- Connecting to development environments (Local files, IDEs, version control)

- Integrating with custom systems (proprietary tools and data sources)

Based on the above, we can see that Skills and MCP address different challenges and work together to complement each other.

| **Dimension**           | **MCP**                                                        | **Skills**                                                              |
| ----------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------- |
| **Core Value**          | Connects to external systems (databases, APIs, SaaS platforms) | Defines behavior specifications (how to process and present data)       |
| **Questions Answered**  | "What can Claude access?"                                      | "What should Claude do?"                                                |
| **Implementation**      | Client-server protocol + JSON Schema                           | Markdown file + YAML metadata                                           |
| **Context Consumption** | Tens of thousands of tokens (multiple server accumulations)    | 30-50 tokens per operation                                              |
| **Use Cases**           | Querying large databases, calling GitHub APIs                  | Defining search strategies, applying filtering rules, output formatting |

Let's take code search as an example.

- **MCP (e.g., claude-context):** Provides the ability to access the Milvus vector database.

- **Skills:** Defines the workflow, such as prioritizing the most recently modified code, sorting results by relevance, and presenting the data in a Markdown table.

MCP provides the capability, while Skills define the process. Together, they form a complementary pair.

## How to Build Custom Skills with Claude-Context and Milvus

[Claude-Context](https://github.com/zilliztech/claude-context) is an MCP plugin that adds semantic code search functionality to Claude Code, turning the entire codebase into Claude’s context.

### Prerequisite

System Requirements:

- **Node.js**: Version >= 20.0.0 and < 24.0.0

- **OpenAI API Key** (for embedding models)

- [**Zilliz Cloud**](https://zilliz.com.cn/) **API Key** (managed Milvus service)

### Step 1: Configure the MCP Service (claude-context)

Run the following command in the terminal:

```
claude mcp add claude-context \
  -e OPENAI_API_KEY=sk-your-openai-api-key \
  -e MILVUS_ADDRESS=https://xxxxxxxxx-cn-hangzhou.cloud.zilliz.com.cn \
  -e MILVUS_TOKEN=your-zilliz-cloud-api-key \
  -e COLLECTION_NAME=medium_articles \
  -- npx @zilliz/claude-context-mcp@latest
```

Check the Configuration:

```
claude mcp list
```

![](https://assets.zilliz.com/Check_the_Configuration_5d2abd5ee8.webp)

The MCP setup is complete. Claude can now access the Milvus vector database.

### Step 2: Create the Skill

Create the Skills directory:

```
mkdir -p ~/.claude/skills/milvus-code-search
cd ~/.claude/skills/milvus-code-search
```

Create the SKILL.md file:

```
---
name: milvus-code-search
description: A semantic code search and architecture analysis skill designed for the Milvus codebase
---

## Instructions
When the user asks questions related to the Milvus codebase, I will:

1. **Code Search** : Use semantic search to locate relevant code snippets across the Milvus repository  
2. **Architecture Analysis** : Analyze Milvus’s module structure, component relationships, and design patterns  
3. **Feature Explanation** : Explain how specific features are implemented and how the corresponding logic works  
4. **Development Guidance** : Provide suggestions, best practices, and improvement ideas for modifying the code  

## Target Repository
- **Core Modules**:  
  - `internal/` — Core internal components  
  - `pkg/` — Public packages and utilities  
  - `client/` — Go client implementation  
  - `cmd/` — Command-line tools  

## Usage Examples

### Architecture Query
User: How does Milvus’s query coordinator work?  
Assistant: *(searching for `querycoordv2`)* Let me walk you through how the query coordinator operates in Milvus…

### Feature Implementation
User: How does Milvus implement vector indexing?  
Assistant: *(searching for `index` code)* The vector indexing logic in Milvus is mainly implemented in the following modules…

### Code Understanding
User: What does this function do? *(points to a specific file)*  
Assistant: *(analyzing the surrounding code)* Based on the context of the Milvus codebase, this function is responsible for…

### Development Guidance
User: How can I add a new vector distance metric to Milvus?  
Assistant: *(searching for `distance` implementations)* Following the existing pattern, you can add a new distance method by…

## Best Practices
1. **Precise Search** : Use specific technical terms and module names  
2. **Contextual Understanding** : Interpret code within Milvus’s overall system architecture  
3. **Actionable Advice** : Provide practical, implementation-ready suggestions  
4. **Performance Awareness** : Consider Milvus’s requirements as a high-performance vector database  

---

*A custom code-search Skill tailored for the open-source Milvus vector database project.*

---
```

### Step 3: Restart Claude to Apply Skills

Run the following command to restart Claude:

```
claude
```

**Note:** After the configuration is complete, you can immediately use the Skills to query the Milvus codebase.

Below is an example of how it works. 

Query: How does Milvus QueryCoord work?

![](https://assets.zilliz.com/code_a95429ddb0.png)

![](https://assets.zilliz.com/code2_d58a942777.png)

![](https://assets.zilliz.com/code3_6c9f835c65.png)

## Conclusion

At its core, Skills act as a mechanism for encapsulating and transferring specialized knowledge. By using Skills, AI can inherit a team's experience and follow industry best practices—whether that’s a checklist for code reviews or documentation standards. When this tacit knowledge is made explicit through Markdown files, the quality of AI-generated outputs can see a significant improvement.

Looking ahead, the ability to leverage Skills effectively could become a key differentiator in how teams and individuals use AI to their advantage. 

As you explore the potential of AI in your organization, Milvus stands as a critical tool for managing and searching large-scale vector data. By pairing Milvus’ powerful vector database with AI tools like Skills, you can improve not only your workflows but also the depth and speed of your data-driven insights.

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) to chat with our engineers and other AI engineers in the community. You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Exploring Skills, MCP, and Milvus for Smarter AI Workflows

Is MCP Already Outdated? The Real Reason Anthropic Shipped Skills—and How to Pair Them with Milvus



*This post was originally published on [Cloudian](https://cloudian.com/blog/cloudian-deploys-new-nvidia-rdma-for-s3-compatible-storage-a-game-changer-for-ai-storage-performance/) and is reposted here with permission.*

Cloudian has collaborated with NVIDIA to add support for RDMA for S3-compatible storage to its HyperStore® solution, drawing on its 13+ years of experience in S3 API implementation. As an S3-API based platform with parallel processing architecture, Cloudian is uniquely suited to both contribute to and capitalize on the development of this technology. This collaboration leverages Cloudian’s deep expertise in object storage protocols and NVIDIA’s leadership in compute and network acceleration to create a solution that seamlessly integrates high-performance computing with enterprise-scale storage.

NVIDIA has announced the upcoming general availability of RDMA for S3-compatible storage (Remote Direct Memory Access) technology, marking a significant milestone in AI infrastructure evolution. This breakthrough technology promises to transform how organizations handle the massive data requirements of modern AI workloads, delivering unprecedented performance improvements while maintaining the scalability and simplicity that has made S3-compatible object storage the foundation of cloud computing.

## What is RDMA for S3-compatible storage?

This launch represents a fundamental advancement in how storage systems communicate with AI accelerators. The technology enables direct data transfers between S3 API-compatible object storage and GPU memory, completely bypassing traditional CPU-mediated data paths. Unlike conventional storage architectures that route all data transfers through the CPU and system memory—creating bottlenecks and latency—RDMA for S3-compatible storage establishes a direct highway from storage to GPU.

![](https://assets.zilliz.com/RDMA_for_S3_compatible_storage_18fdd69d02.webp)

At its core, this technology eliminates intermediate steps with a direct pathway that reduces latency, dramatically cuts CPU processing demands, and significantly reduces power consumption. The result is storage systems that can deliver data at the speed modern GPUs require for demanding AI applications.

The technology maintains compatibility with the ubiquitous S3 APIs while adding this high-performance data path. Commands are still issued through standard S3-API based storage protocols, but the actual data transfer occurs via RDMA directly to GPU memory, bypassing the CPU entirely and eliminating the overhead of TCP protocol processing.

## Breakthrough Performance Results

The performance improvements delivered by RDMA for S3-compatible storage are nothing short of transformational. Real-world testing demonstrates the technology’s ability to eliminate storage I/O bottlenecks that constrain AI workloads.

### Dramatic Speed Improvements:

- **35 GB/s per node throughput** (reads) measured, with linear scalability across clusters

- **Scalability to TBs/s** with Cloudian’s parallel processing architecture

- **3-5x throughput improvement** compared to conventional TCP-based object storage

### Resource Efficiency Gains:

- **90% reduction in CPU utilization** by establishing direct data pathways to GPUs

- **Increase GPU utilization** with bottlenecks eliminated

- Dramatic reduction in power consumption through reduced processing overhead

- Cost reductions for AI storage

### 8X Performance Boost on Milvus by Zilliz Vector DB

These performance improvements are particularly evident in vector database operations, where collaboration between Cloudian and Zilliz using [NVIDIA cuVS](https://developer.nvidia.com/cuvs) and [NVIDIA L40S GPUs](https://www.nvidia.com/en-us/data-center/l40s/) demonstrated an **8x performance boost in Milvus operations** when compared with CPU-based systems and TCP-based data transfer. This represents a fundamental shift from storage being a constraint to storage enabling AI applications to achieve their full potential.

![](https://assets.zilliz.com/vector_indexing_time_354bdd4b46.webp)

## Why S3 API-based Object Storage for AI Workloads

The convergence of RDMA technology  with object storage architecture creates the ideal foundation for AI infrastructure, addressing multiple challenges that have constrained traditional storage approaches.

**Exabyte Scalability for AI’s Data Explosion:** AI workloads, particularly those involving synthetic and multi-modal data, are pushing storage requirements into the 100-petabyte range and beyond. Object storage’s flat address space scales seamlessly from petabytes to exabytes, accommodating the exponential growth in AI training datasets without the hierarchical limitations that constrain file-based systems.

**Unified Platform for Complete AI Workflows:** Modern AI operations span data ingestion, cleansing, training, checkpointing, and inference—each with distinct performance and capacity requirements. S3-compatible object storage supports this entire spectrum through consistent API access, eliminating the complexity and cost of managing multiple storage tiers. Training data, models, checkpoint files, and inference datasets can all reside in a single, high-performance data lake.

**Rich Metadata for AI Operations:** Critical AI operations like search and enumeration are fundamentally metadata-driven. Object storage’s rich, customizable metadata capabilities enable efficient data tagging, searching, and management—essential for organizing and retrieving data in complex AI model training and inference workflows.

**Economic and Operational Advantages:** S3-compatible object storage delivers up to 80% lower total cost of ownership compared to file storage alternatives, leveraging industry-standard hardware and independent scaling of capacity and performance. This economic efficiency becomes crucial as AI datasets reach enterprise scale.

**Enterprise Security and Governance:** Unlike GPUDirect implementations that require kernel-level modifications, RDMA for S3-compatible storage requires no vendor-specific kernel changes, maintaining system security and regulatory compliance. This approach is particularly valuable in sectors like healthcare and finance where data security and regulatory compliance are paramount.

## The Road Ahead

NVIDIA’s announcement of RDMA for S3-compatible storage general availability represents more than a technological milestone—it signals the maturation of AI infrastructure architecture. By combining the limitless scalability of object storage with the breakthrough performance of direct GPU access, organizations can finally build AI infrastructures that scale with their ambitions.

As AI workloads continue to grow in complexity and scale, RDMA for S3-compatible storage provides the storage foundation that enables organizations to maximize their AI investments while maintaining data sovereignty and operational simplicity. The technology transforms storage from a bottleneck into an enabler, allowing AI applications to achieve their full potential at enterprise scale.

For organizations planning their AI infrastructure roadmap, the general availability of RDMA for S3-compatible storage marks the beginning of a new era where storage performance truly matches the demands of modern AI workloads.

## Industry Perspectives

As AI becomes increasingly central to healthcare delivery, we continuously seek to boost the performance and efficiency of our infrastructure. The new RDMA for S3-compatible storage from NVIDIA and Cloudian will be critical for our medical imaging analysis and diagnostic AI applications, where processing large datasets quickly can directly impact patient care, while reducing costs of moving data between S3-API based storage devices and SSD based NAS storages.  – _Dr. Swapnil Rane MD, DNB, PDCC (Nephropath), Mres (TCM), Fellowship in Oncopath, FRCPath Professor (F) of Pathology, PI, AI/Computational Pathology And Imaging Lab OIC- Department of Digital and Computational Oncology, Tata Memorial Centre_

“NVIDIA’s RDMA for S3-compatible announcement confirms the value of our Cloudian-based AI infrastructure strategy. We enable organizations to run high-performance AI at scale while preserving S3 API compatibility that keeps migration simple and application development costs low.” – _Sunil Gupta, Co-founder, Managing Director & Chief Executive Officer (CEO), Yotta Data Services_

“As we expand our on-premises capabilities to deliver sovereign AI, NVIDIA’s RDMA for S3-compatible storage technology and Cloudian’s high-performance object storage give us the performance we need without compromising data residency and without requiring any kernel-level modifications. The Cloudian HyperStore platform lets us scale to exabytes while keeping our sensitive AI data completely under our control.” – _Logan Lee, EVP & Head of Cloud, Kakao_

“We’re excited about NVIDIA’s announcement of the upcoming RDMA for S3-compatible storage GA release. Our testing with Cloudian showed up to 8X performance improvement for vector database operations, which will let our Milvus by Zilliz users achieve cloud-scale performance for demanding AI workloads while maintaining complete data sovereignty.” – _Charles Xie, Founder and CEO of Zilliz_


8× Milvus Boost with Cloudian HyperStore and NVIDIA RDMA

Unlocking 8× Milvus Performance with Cloudian HyperStore and NVIDIA RDMA for S3 Storage



*This post was originally published on [HPE Community](https://community.hpe.com/t5/around-the-storage-block/power-high-performance-rag-for-genai-with-hpe-alletra-storage-mp/ba-p/7257369) and is reposted here with permission.*

HPE Alletra Storage MP X10000 and Milvus power scalable, low-latency RAG, enabling LLMs to deliver accurate, context-rich responses with high-performance vector search for GenAI workloads.

## In generative AI, RAG needs more than just an LLM

Context unleashes the true power of generative AI (GenAI) and large language models (LLMs). When an LLM has the right signals to orient its responses, it can deliver answers that are accurate, relevant, and trustworthy.

Think of it this way: if you were dropped into a dense jungle with a GPS device but no satellite signal. The screen shows a map, but without your current position, it is useless for navigation. On the contrary, a GPS with a strong satellite signal doesn’t just show a map; it gives you turn-by-turn guidance.

That’s what retrieval-augmented generation (RAG) does for LLMs. The model already has the map (its pretrained knowledge), but not the direction (your domain-specific data). LLMs without RAG are like GPS devices that are full of knowledge but have no real-time orientation. RAG provides the signal that tells the model where it is and where to go.

RAG grounds model responses in trusted, up-to-date knowledge pulled from your own domain-specific content from policies, product docs, tickets, PDFs, code, audio transcripts, images, and more. Making RAG work at scale is challenging. The retrieval process needs to be fast enough to keep user experiences seamless, accurate enough to return the most relevant information, and predictable even when the system is under heavy load. That means handling high query volumes, ongoing data ingestion, and background tasks like index building without performance degradation. Spinning up a RAG pipeline with a few PDFs is relatively straightforward. However, when scaling to hundreds of PDFs, it becomes significantly more challenging. You can’t keep everything in memory, so a robust and efficient storage strategy becomes essential to manage embeddings, indexes, and retrieval performance. RAG requires a vector database and a storage layer that can keep pace as concurrency and data volumes grow.

## Vector databases power RAG

At the heart of RAG is semantic search, finding information by meaning rather than exact keywords. This is where vector databases come in. They store high-dimensional embeddings of text, images, and other unstructured data, enabling similarity search that retrieves the most relevant context for your queries. Milvus is a leading example: a cloud-native, open-source vector database built for billion-scale similarity search. It supports hybrid search, combining vector similarity with keyword and scalar filters for precision, and offers independent scaling of compute and storage with GPU-aware optimization options for acceleration. Milvus also manages data through a smart segment lifecycle, moving from growing to sealed segments with compaction and multiple approximate nearest neighbor (ANN) indexing options such as HNSW and DiskANN, ensuring performance and scalability for real-time AI workloads such as RAG.

## The hidden challenge: Storage throughput & latency

Vector search workloads put pressure on every part of the system. They demand high‑concurrency ingestion while maintaining low‑latency retrieval for interactive queries. At the same time, background operations such as index building, compaction, and data reloads must run without disrupting live performance. Many performance bottlenecks in traditional architectures trace back to storage. Whether it’s input/output (I/O) limitations, metadata lookup delays, or concurrency constraints. To deliver predictable, real‑time performance at scale, the storage layer must keep pace with the demands of vector databases.

## The storage foundation for high performance vector search

[HPE Alletra Storage MP X10000](https://www.hpe.com/in/en/alletra-storage-mp-x10000.html) is a flash-optimized, all-NVMe, S3-compatible object storage platform engineered for real-time performance at scale. Unlike traditional capacity-focused object stores, HPE Alletra Storage MP X10000 is designed for low-latency, high-throughput workloads like vector search. Its log-structured key-value engine and extent-based metadata enable highly parallel reads and writes, while GPUDirect RDMA provides zero-copy data paths that reduce CPU overhead and accelerate data movement to GPUs. The architecture supports disaggregated scaling, allowing capacity and performance to grow independently, and includes enterprise-grade features such as encryption, role-based access control (RBAC), immutability, and data durability. Combined with its cloud-native design, HPE Alletra Storage MP X10000 integrates seamlessly with Kubernetes environments, making it an ideal storage foundation for Milvus deployments.

## HPE Alletra Storage MP X10000 and Milvus: A scalable foundation for RAG

HPE Alletra Storage MP X10000 and Milvus complement each other to deliver RAG that is fast, predictable, and easy to scale. Figure 1 illustrates the architecture of scalable AI use cases and RAG pipelines, showing how Milvus components deployed in a containerized environment interact with high performance object storage from HPE Alletra Storage MP X10000.

![](https://assets.zilliz.com/Figure_1_Architecture_of_scalable_AI_use_cases_and_RAG_pipeline_using_HPE_Alletra_Storage_MP_X10000_and_Milvus_ed3a87a5ee.png)

Milvus cleanly separates compute from storage, while HPE Alletra Storage MP X10000 provides high‑throughput, low‑latency object access that keeps pace with vector workloads. Together, they enable predictable scale‑out performance: Milvus distributes queries across shards, and HPE Alletra Storage MP X10000’s fractional, multidimensional scaling keeps latency consistent as data and QPS grow. In simple terms, you add exactly the capacity or performance you need, when you need it. Operational simplicity is another advantage: HPE Alletra Storage MP X10000 sustains maximum performance from a single bucket, eliminating complex tiering, while enterprise features (encryption, RBAC, immutability, robust durability) support on‑prem or hybrid deployments with strong data sovereignty and consistent service-level objectives (SLOs).

When vector search scales, storage is often blamed for slow ingestion, compaction, or retrieval. With Milvus on HPE Alletra Storage MP X10000, that narrative changes. The platform’s all‑NVMe, log‑structured architecture and GPUDirect RDMA option deliver consistent, ultra‑low‑latency object access—even under heavy concurrency and during lifecycle operations like index build and reload. In practice, your RAG pipelines remain compute‑bound, not storage‑bound. As collections grow and query volumes spike, Milvus stays responsive while HPE Alletra Storage MP X10000 preserves I/O headroom, enabling predictable, linear scalability without re‑architecting storage. This becomes especially important as RAG deployments scale beyond initial proof-of-concept stages and move into full production.

## Enterprise-ready RAG: Scalable, predictable, and built for GenAI

For RAG and real-time GenAI workloads, the combination of HPE Alletra Storage MP X10000 and Milvus delivers a future-ready foundation that scales with confidence. This integrated solution empowers organizations to build intelligent systems that are fast, elastic, and secure—without compromising on performance or manageability. Milvus provides distributed, GPU-accelerated vector search with modular scaling, while HPE Alletra Storage MP X10000 ensures ultrafast, low-latency object access with enterprise-grade durability and lifecycle management. Together, they decouple compute from storage, enabling predictable performance even as data volumes and query complexity grow. Whether you’re serving real-time recommendations, powering semantic search, or scaling across billions of vectors, this architecture keeps your RAG pipelines responsive, cost-efficient, and cloud-optimized. With seamless integration into Kubernetes and HPE GreenLake cloud, you gain unified management, consumption-based pricing, and the flexibility to deploy across hybrid or private cloud environments. HPE Alletra Storage MP X10000 and Milvus: a scalable, high performance RAG solution built for the demands of modern GenAI.


Optimized RAG with HPE Alletra Storage MP X10000 + Milvus

Power high performance RAG for GenAI with HPE Alletra Storage MP + Milvus


How Parlant × Milvus Brings Control to LLM Agent Behavior

Beyond Context Overload: How Parlant × Milvus Brings Control and Clarity to LLM Agent Behavior



Building an AI workflow often feels harder than it should. Between writing glue code, debugging API calls, and managing data pipelines, the process can eat up hours before you even see results. [**Langflow**](https://www.langflow.org/) and [**Milvus**](https://milvus.io/) simplify this dramatically — giving you a code-light way to design, test, and deploy retrieval-augmented generation (RAG) workflows in minutes, not days.

**Langflow** offers a clean, drag-and-drop interface that feels more like sketching ideas on a whiteboard than coding. You can visually connect language models, data sources, and external tools to define your workflow logic — all without touching a line of boilerplate code.

Paired with **Milvus**, the open-source vector database that gives LLMs long-term memory and contextual understanding, the two form a complete environment for production-grade RAG. Milvus efficiently stores and retrieves embeddings from your enterprise or domain-specific data, allowing LLMs to generate answers that are grounded, accurate, and context-aware.

In this guide, we’ll walk through how to combine Langflow and Milvus to build an advanced RAG workflow — all through a few drags, drops, and clicks.


## What is Langflow? 

Before going through the RAG demo, let’s learn what Langflow is and what it can do. 

Langflow is an open-source, Python-based framework that makes it easier to build and experiment with AI applications. It supports key AI capabilities such as agents and the Model Context Protocol (MCP), giving developers and non-developers alike a flexible foundation for creating intelligent systems. 

At its core, Langflow provides a visual editor. You can drag, drop, and connect different resources to design complete applications that combine models, tools, and data sources. When you export a workflow, Langflow automatically generates a file named `FLOW_NAME.json` on your local machine. This file records all the nodes, edges, and metadata that describe your flow, allowing you to version-control, share, and reproduce projects easily across teams.

![](https://assets.zilliz.com/Langflow_s_visual_editor_cd553ad4ad.png)

Behind the scenes, a Python-based runtime engine executes the flow. It orchestrates LLMs, tools, retrieval modules, and routing logic — managing data flow, state, and error handling to ensure smooth execution from start to finish.

Langflow also includes a rich component library with prebuilt adapters for popular LLMs and vector databases — including [Milvus](https://milvus.io/). You can extend this further by creating custom Python components for specialized use cases. For testing and optimization, Langflow offers step-by-step execution, a Playground for rapid testing, and integrations with LangSmith and Langfuse for monitoring, debugging, and replaying workflows end-to-end.


## Hands-on Demo: How to Build a RAG Workflow with Langflow and Milvus

Building on Langflow’s architecture, Milvus can serve as the vector database that manages embeddings and retrieves private enterprise data or domain-specific knowledge.

In this demo, we’ll use Langflow’s Vector Store RAG template to demonstrate how to integrate Milvus and build a vector index from local data, enabling efficient, context-enhanced question answering.

![](https://assets.zilliz.com/data_processing_flow_289a9376c9.webp)

### Prerequisites：

1.Python 3.11 (or Conda)

2.uv

3.Docker & Docker Compose

4.OpenAI key


### Step 1. Deploy Milvus Vector Database

Download the deployment files.
```
wget <https://github.com/Milvus-io/Milvus/releases/download/v2.5.12/Milvus-standalone-docker-compose.yml> -O docker-compose.yml
```
Start the Milvus service.
```
docker-compose up -d
```
```
docker-compose ps -a
```

![](https://assets.zilliz.com/start_milvus_service_860353ed55.webp)

### Step 2. Create a Python Virtual Environment
```
conda create -n langflow
# activate langflow and launch it
conda activate langflow
```
### Step 3. Install the Latest Packages
```
pip install langflow -U
```
### Step 4. Launch Langflow
```
uv run langflow run
```
Visit Langflow.
```
<http://127.0.0.1:7860>
```
### Step 5. Configure the RAG Template
Select the Vector Store RAG template in Langflow.

![](https://assets.zilliz.com/rag1_fcb0d1c3c5.webp)

![](https://assets.zilliz.com/rag2_f750e10a41.webp)

Choose Milvus as your default vector database.

![](https://assets.zilliz.com/vdb_milvus_925c6ce846.webp)

In the left panel, search for “Milvus” and add it to your flow.

![](https://assets.zilliz.com/add_milvus1_862d14d0d0.webp)

![](https://assets.zilliz.com/add_milvus2_4e3d6aacda.webp)

Configure Milvus connection details. Leave other options as the default for now.

![](https://assets.zilliz.com/connect1_a27d3e4f43.webp)

![](https://assets.zilliz.com/connect2_d8421c1525.webp)

Add your OpenAI API key to the relevant node.

![](https://assets.zilliz.com/openai_key_7a6596868c.webp)

![](https://assets.zilliz.com/openai_key2_4753bfb4d0.webp)

### Step 6. Prepare Test Data
Note: Use the official FAQ for Milvus 2.6 as the test data.
```
https://github.com/milvus-io/milvus-docs/blob/v2.6.x/site/en/faq/product_faq.md
```
### Step 7. Phase One Testing
Upload your dataset and ingest it into Milvus.
Note: Langflow then converts your text into vector representations. You must upload at least two datasets, or the embedding process will fail. This is a known bug in Langflow’s current node implementation.

![](https://assets.zilliz.com/ingest_7b804d870a.webp)

![](https://assets.zilliz.com/ingest2_fc7f1e4d9a.webp)

Check the status of your nodes.

![](https://assets.zilliz.com/test_48e02d48ca.webp)

### Step 8. Phase Two Testing

![](https://assets.zilliz.com/ingest_7b804d870a.webp)

### Step 9. Run the Full RAG Workflow

![](https://assets.zilliz.com/full_flow1_5b4f4962f5.webp)

![](https://assets.zilliz.com/full_flow2_535c722a3d.webp)

## Conclusion

Building AI workflows doesn’t have to be complicated. Langflow + Milvus makes it fast, visual, and code-light — a simple way to enhance RAG without heavy engineering effort.

Langflow’s drag-and-drop interface makes it a suitable choice for teaching, workshops, or live demos, where you need to demonstrate how AI systems work in a clear and interactive manner. For teams seeking to integrate intuitive workflow design with enterprise-grade vector retrieval, combining Langflow’s simplicity with Milvus’s high-performance search delivers both flexibility and power.

👉 Start building smarter RAG workflows with [Milvus](https://milvus.io/) today.

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Drag, Drop, and Deploy RAG Workflows with Langflow & Milvus

Drag, Drop, and Deploy: How to Build RAG Workflows with Langflow and Milvus



In a vector database, we often need to quickly find the most similar results among vast collections of high-dimensional vectors—such as image features, text embeddings, or audio representations. Without an index, the only option is to compare the query vector against every single vector in the dataset. This **brute-force search** might work when you have a few thousand vectors, but once you’re dealing with tens or hundreds of millions, it becomes unbearably slow and computationally expensive. 

That’s where **Approximate Nearest Neighbor (ANN)** search comes in. Think of it like looking for a specific book in a massive library. Instead of checking every book one by one, you start by browsing the sections most likely to contain it. You might not get the _exact_ same results as a full search, but you’ll get very close—and in a fraction of the time. In short, ANN trades a slight loss in accuracy for a significant boost in speed and scalability.

Among the many ways to implement ANN search, **IVF (Inverted File)** and **HNSW (Hierarchical Navigable Small World)** are two of the most widely used. But IVF stands out for its efficiency and adaptability in large-scale vector search. In this article, we’ll walk you through how IVF works and how it compares with HNSW—so you can understand their trade-offs and choose the one that best fits your workload.

## What is an IVF Vector Index?

**IVF (Inverted File)** is one of the most widely used algorithms for ANN. It borrows its core idea from the “inverted index” used in text retrieval systems—only this time, instead of words and documents, we’re dealing with vectors in a high-dimensional space.

Think of it like organizing a massive library. If you dumped every book (vector) into one giant pile, finding what you need would take forever. IVF solves this by first **clustering** all vectors into groups, or _buckets_. Each bucket represents a “category” of similar vectors, defined by a **centroid**—a kind of summary or “label” for everything inside that cluster.

When a query comes in, the search happens in two steps:

**1. Find the nearest clusters.** The system looks for the few buckets whose centroids are closest to the query vector—just like heading straight to the two or three library sections most likely to have your book.

**2. Search within those clusters.** Once you’re in the right sections, you only need to look through a small set of books instead of the entire library.

This approach cuts down the amount of computation by orders of magnitude. You still get highly accurate results—but much faster. 


## How to Build an IVF Vector Index

The process of building an IVF vector index involves three main steps: K-means Clustering, Vector Assignment, and Compression Encoding (Optional). The full process looks like this:

![](https://assets.zilliz.com/ivf_building_process_90c2966975.webp)

### Step 1: K-means Clustering

First, run k-means clustering on the dataset X to divide the high-dimensional vector space into nlist clusters. Each cluster is represented by a centroid, which is stored in the centroid table C. The number of centroids, nlist, is a key hyperparameter that determines how fine-grained the clustering will be.

Here’s how k-means works under the hood:

- **Initialization:** Randomly select _nlist_ vectors as the initial centroids.

- **Assignment:** For each vector, compute its distance to all centroids and assign it to the nearest one.

- **Update:** For each cluster, calculate the average of its vectors and set that as the new centroid.

- **Iteration and convergence:** Repeat assignment and update until centroids stop changing significantly or a maximum number of iterations is reached.

Once k-means converges, the resulting nlist centroids form the “index directory” of IVF. They define how the dataset is coarsely partitioned, allowing queries to quickly narrow down the search space later on.

Think back to the library analogy: training centroids is like deciding how to group books by topic:

- A larger nlist means more sections, each with fewer, more specific books.

- A smaller nlist means fewer sections, each covering a broader, more mixed range of topics.

### Step 2: Vector Assignment

Next, each vector is assigned to the cluster whose centroid it’s closest to, forming inverted lists (List_i). Each inverted list stores the IDs and storage information of all vectors that belong to that cluster. 

You can think of this step like shelving the books into their respective sections. When you’re looking for a title later, you only need to check the few sections that are most likely to have it, instead of wandering the whole library.

### Step 3: Compression Encoding (Optional)

To save memory and speed up computation, vectors within each cluster can go through compression encoding. There are two common approaches:

- **SQ8 (Scalar Quantization):** This method quantizes each dimension of a vector into 8 bits. For a standard `float32` vector, each dimension typically takes up 4 bytes. With SQ8, it’s reduced to just 1 byte—achieving a 4:1 compression ratio while keeping the vector’s geometry largely intact.

- **PQ (Product Quantization):** It splits a high-dimensional vector into several subspaces. For example, a 128-dimensional vector can be divided into 8 sub-vectors of 16 dimensions each. In each subspace, a small codebook (typically with 256 entries) is pre-trained, and each sub-vector is represented by an 8-bit index pointing to its nearest codebook entry. This means the original 128-D `float32` vector (which requires 512 bytes) can be represented using only 8 bytes (8 subspaces × 1 byte each), achieving a 64:1 compression ratio.

## How to Use the IVF Vector Index for Search

Once the centroid table, inverted lists, and the compression encoder and codebooks (optional) are built, the IVF index can be used to accelerate similarity search. The process typically has three main steps, as shown below:

![](https://assets.zilliz.com/ivf_search_process_025d3f444f.webp)

### Step 1: Calculate distances from the query vector to all centroids

When a query vector q arrives, the system first determines which clusters it is most likely to belong to. Then, it computes the distance between q and every centroid in the centroid table C—usually using Euclidean distance or inner product as the similarity metric. The centroids are then sorted by their distance to the query vector, producing an ordered list from nearest to farthest.

For example, as shown in the illustration, the order is: C4 < C2 < C1 < C3 < C5.


### Step 2: Select the nearest nprobe clusters

To avoid scanning the entire dataset, IVF only searches within the top _nprobe_ clusters that are closest to the query vector.

The parameter nprobe defines the search scope and directly affects the balance between speed and recall:

- A smaller nprobe leads to faster queries but may reduce recall.

- A larger nprobe improves recall but increases latency.

In real-world systems, nprobe can be dynamically tuned based on the latency budget or accuracy requirements.
In the example above, if nprobe = 2, the system will only search within Cluster 2 and Cluster 4—the two nearest clusters.


### Step 3: Search the nearest neighbor in the selected clusters

Once the candidate clusters are selected, the system compares the query vector q with the vectors stored inside them.
&#x20;There are two main modes of comparison:

- **Exact Comparison (IVF_FLAT)**: The system retrieves the original vectors from the selected clusters and computes their distances to q directly, returning the most accurate results.

- **Approximate Comparison (IVF_PQ / IVF_SQ8)**: When PQ or SQ8 compression is used, the system employs a **lookup table method** to accelerate distance computation. Before the search begins, it precomputes the distances between the query vector and each codebook entry. Then, for each vector, it can simply “look up and sum” these precomputed distances to estimate similarity.

Finally, the candidate results from all searched clusters are merged and re-ranked, producing the Top-k most similar vectors as the final output.


## IVF In-Practice

Once you understand how IVF vector indexes are **built** and **searched**, the next step is to apply them for real-world workloads. In practice, you’ll often need to balance **performance**, **accuracy**, and **memory usage**. Below are some practical guidelines drawn from engineering experience.


### How to Choose the Right nlist

As mentioned earlier, the parameter nlist determines the number of clusters into which the dataset is divided when building an IVF index. 

- **Larger nlist**: Creates finer-grained clusters, meaning each cluster contains fewer vectors. This reduces the number of vectors scanned during search and generally results in faster queries. But building the index takes longer, and the centroid table consumes more memory.

- **Smaller nlist**: Speeds up index construction and reduces memory usage, but each cluster becomes more “crowded.” Each query must scan more vectors within a cluster, which can lead to performance bottlenecks.

Based on these trade-offs, here’s a practical rule of thumb: 

For datasets at the **million scale**, a good starting point is **nlist ≈ √n** (n is the number of vectors in the data shard being indexed).

For example, if you have 1 million vectors, try nlist = 1,000. For larger datasets—tens or hundreds of millions—most vector databases shard the data so that each shard contains around one million vectors, keeping this rule practical.

Because nlist is fixed at index creation, changing it later means rebuilding the entire index. So it’s best to experiment early. Test several values—ideally in powers of two (e.g., 1024, 2048)—to find the sweet spot that balances speed, accuracy, and memory for your workload.


### How to Tune nprobe

The parameter nprobe controls the number of clusters searched during query time. It directly affects the trade-off between recall and latency.

- **Larger nprobe**: Covers more clusters, leading to higher recall but also higher latency. The delay generally increases linearly with the number of clusters searched.

- **Smaller nprobe**: Searches fewer clusters, resulting in lower latency and faster queries. However, it may miss some true nearest neighbors, slightly lowering recall and result accuracy.

If your application is not extremely sensitive to latency, it’s a good idea to experiment with nprobe dynamically—for example, testing values from 1 to 16 to observe how recall and latency change. The goal is to find the sweet spot where recall is acceptable and latency remains within your target range.

Since nprobe is a runtime search parameter, it can be adjusted on the fly without requiring the index to be rebuilt. This enables fast, low-cost, and highly flexible tuning across different workloads or query scenarios.

### Common Variants of the IVF Index

When building an IVF index, you’ll need to decide whether to use compression encoding for the vectors in each cluster—and if so, which method to use.

This results in three common IVF index variants: 

| **IVF Variant** | **Key Features**                                                                                                                                      | **Use Cases**                                                                                                                                                                                                                               |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **IVF_FLAT**   | Stores raw vectors within each cluster without compression. Offers the highest accuracy, but also consumes the most memory.                           | Ideal for medium-scale datasets (up to hundreds of millions of vectors) where high recall (95%+) is required.                                                                                                                               |
| **IVF_PQ**     | Applies Product Quantization (PQ) to compress vectors within clusters. By adjusting the compression ratio, memory usage can be significantly reduced. | Suitable for large-scale vector search (hundreds of millions or more) where some accuracy loss is acceptable. With a 64:1 compression ratio, recall is typically around 70%, but can reach 90% or higher by lowering the compression ratio. |
| **IVF_SQ8**    | Uses Scalar Quantization (SQ8) to quantize vectors. Memory usage sits between IVF_FLAT and IVF_PQ.                                                  | Ideal for large-scale vector search where you need to maintain relatively high recall (90%+) while improving efficiency.                                                                                                                    |


## IVF vs HNSW: Pick What Fits

Besides IVF, **HNSW (Hierarchical Navigable Small World)** is another widely used in-memory vector index. The table below highlights the key differences between the two.

|                                  | **IVF**                                                                                                                        | **HNSW**                                                                                                                                                                      |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Algorithm Concept**            | Clustering and bucketing                                                                                                       | Multi-layer graph navigation                                                                                                                                                  |
| **Memory Usage**                 | Relatively low                                                                                                                 | Relatively high                                                                                                                                                               |
| **Index Build Speed**            | Fast (only requires clustering)                                                                                                | Slow (needs multi-layer graph construction)                                                                                                                                   |
| **Query Speed (No Filtering)**   | Fast, depends on _nprobe_                                                                                                      | Extremely fast, but with logarithmic complexity                                                                                                                               |
| **Query Speed (With Filtering)** | Stable — performs coarse filtering at the centroid level to narrow down candidates                                             | Unstable — especially when the filtering ratio is high (90%+), the graph becomes fragmented and may degrade to near full-graph traversal, even slower than brute-force search |
| **Recall Rate**                  | Depends on whether compression is used; without quantization, recall can reach **95%+**                                        | Usually higher, around **98%+**                                                                                                                                               |
| **Key Parameters**               | _nlist_, _nprobe_                                                                                                              | _m_, _ef_construction_, _ef_search_                                                                                                                                         |
| **Use Cases**                    | When memory is limited, but high query performance and recall are required; well-suited for searches with filtering conditions | When memory is sufficient and the goal is extremely high recall and query performance, but filtering is not needed, or the filtering ratio is low                             |

In real-world applications, it’s very common to include filtering conditions—for example, “only search vectors from a specific user” or “limit results to a certain time range.” Due to differences in their underlying algorithms, IVF generally handles filtered searches more efficiently than HNSW.

The strength of IVF lies in its two-level filtering process. It can first perform a coarse-grained filter at the centroid (cluster) level to quickly narrow down the candidate set, and then conduct fine-grained distance calculations within the selected clusters. This maintains stable and predictable performance, even when a large portion of the data is filtered out.

In contrast, HNSW is based on graph traversal. Because of its structure, it cannot directly leverage filtering conditions during traversal. When the filtering ratio is low, this doesn’t cause major issues. However, when the filtering ratio is high (e.g., more than 90% of data is filtered out), the remaining graph often becomes fragmented, forming many “isolated nodes.” In such cases, the search may degrade into a near full-graph traversal—sometimes even worse than a brute-force search.

In practice, IVF indexes are already powering many high-impact use cases across different domains:

- **E-commerce search:** A user can upload a product image and instantly find visually similar items from millions of listings.

- **Patent retrieval:** Given a short description, the system can locate the most semantically related patents from a massive database—far more efficient than traditional keyword search.

- **RAG knowledge bases:** IVF helps retrieve the most relevant context from millions of tenant documents, ensuring AI models generate more accurate and grounded responses.


## Conclusion

To choose the right index, it all comes down to your specific use case. If you’re working with large-scale datasets or need to support filtered searches, IVF can be the better fit. Compared with graph-based indexes like HNSW, IVF delivers faster index builds, lower memory usage, and a strong balance between speed and accuracy.

[Milvus](https://milvus.io/), the most popular open-source vector database, provides full support for the entire IVF family, including IVF_FLAT, IVF_PQ, and IVF_SQ8. You can easily experiment with these index types and find the setup that best fits your performance and memory needs. For a complete list of indexes Milvus supports, check out this [Milvus Index doc page](https://milvus.io/docs/index-explained.md). 

If you’re building image search, recommendation systems, or RAG knowledge bases, give IVF indexing in Milvus a try—and see how efficient, large-scale vector search feels in action.

How to Choose Between IVF and HNSW for ANN Vector Search

Understanding IVF Vector Index: How It Works and When to Choose It Over HNSW



This post was originally published on [Medium](https://medium.com/opus-engineering/how-opussearch-built-exact-matching-for-enterprise-rag-with-milvus-bm25-aa1098a9888b) and is reposted here with permission.

## The Semantic Search Blind Spot

Picture this: You’re a video editor on deadline. You need clips from “episode 281” of your podcast. You type it into our search. Our AI-powered semantic search, proud of its intelligence, returns clips from 280, 282, and even suggests episode 218 because the numbers are similar, right?

**Wrong**.

When we launched [OpusSearch](https://www.opus.pro/opussearch) for enterprises in January 2025, we thought semantic search would be enough. Natural language queries like “find funny moments about dating” worked beautifully. Our [Milvus](https://milvus.io/)-powered RAG system was crushing it.

**But then reality hit us in the face with user feedback:**

“I just want clips from episode 281. Why is this so hard?”

“When I search ‘That’s what she said,’ I want EXACTLY that phrase, not ‘that’s what he meant.’”

Turns out that video editors and clippers don’t always want AI to be clever. Sometimes they want software to be **straightforward and correct**.

## Why do we care about Search?

We built an [enterprise search function](https://www.opus.pro/opussearch) because we identified that **monetizing** large video catalogs is the key challenge organizations face. Our RAG-powered platform serves as a **growth agent** that enables enterprises to **search, repurpose, and monetize their entire video libraries**. Read about success case stories from **All The Smoke**, **KFC Radio** and **TFTC** [here](https://www.opus.pro/blog/growing-a-new-youtube-channel-in-90-days-without-creating-new-videos).

## Why We Doubled Down on Milvus (Instead of Adding Another Database)

The obvious solution was to add Elasticsearch or MongoDB for exact matching. However, as a startup, maintaining multiple search systems introduces significant operational overhead and complexity.

![](https://assets.zilliz.com/Strong_community_adoption_with_35k_Git_Hub_stars_fbf773dcdb.webp)

Milvus had recently shipped their full-text search feature, and an evaluation with our own dataset **without any tuning** showed compelling advantages:

- **Superior partial matching accuracy**. For example “drinking story” and “jumping high”, other vector DBs returns sometimes “dining story” and “getting high” which alters the meaning.

- Milvus **returns longer, more comprehensive results** than other databases when queries are general, which is naturally more ideal for our use case.

## Architecture from 5000 feet
![](https://assets.zilliz.com/Milvus_is_the_foundational_vector_database_for_our_Enterprise_RAG_architecture_b3c8ebf39c.webp)
## BM25 + Filtering = Exact Match Magic

Milvus’s full-text search isn’t really about exact matching, but it’s about relevance scoring using BM25 ([Best Matching 25](https://en.wikipedia.org/wiki/Okapi_BM25)), which calculates how relevant a document is to your query. It’s great for “find me something close,” but terrible for “find me exactly this.”

We then **combined BM25’s power with Milvus’s TEXT\_MATCH filtering**. Here’s how it works:

1. **Filter first**: TEXT\_MATCH finds documents containing your exact keywords

2. **Rank second**: BM25 sorts those exact matches by relevance

3. **Win**: You get exact matches, ranked intelligently

Think of it as “give me everything with ‘episode 281’, then show me the best ones first.”

## The Code That Made It Work

### Schema Design

**Important**: We disabled stop words entirely, as terms like “The Office” and “Office” represent distinct entities in our content domain.
```
export function getExactMatchFields(): FieldType[] {
 return [
   {
     name: "id",
     data_type: DataType.VarChar,
     is_primary_key: true,
     max_length: 100,
   },
   {
     name: "text",
     data_type: DataType.VarChar,
     max_length: 1000,
     enable_analyzer: true,
     enable_match: true,  // This is the magic flag
     analyzer_params: {
       tokenizer: 'standard',
       filter: [
         'lowercase',
         {
           type: 'stemmer',
           language: 'english',  // "running" matches "run"
         },
         {
           type: 'stop',
           stop_words: [],  // Keep ALL words (even "the", "a")
         },
       ],
     },
   },
   {
     name: "sparse_vector",
     data_type: DataType.SparseFloatVector,
   },
 ]
}
```
### BM25 Function Setup
```
export const FUNCTIONS: FunctionObject[] = [
 {
   name: 'text_bm25_embedding',
   type: FunctionType.BM25,
   input_field_names: ['text'],
   output_field_names: ['sparse_vector'],
   params: {},
 },
]
```
### Index Config

These bm25\_k1 and bm25\_b parameters were tuned against our production dataset for optimal performance.

**bm25\_k1**: Higher values (up to \~2.0) give more weight to repeated term occurrences, while lower values reduce the impact of term frequency after the first few occurrences.

**bm25\_b**: Values closer to 1.0 heavily penalize longer documents, while values closer to 0 ignore document length entirely.
```
index_params: [
 {
   field_name: 'sparse_vector',
   index_type: 'SPARSE_INVERTED_INDEX',
   metric_type: 'BM25',
   params: {
     inverted_index_algo: 'DAAT_MAXSCORE',
     bm25_k1: 1.2,  // How much does term frequency matter?
     bm25_b: 0.75,  // How much does document length matter?
   },
 },
],
```

### The Search Query That Started Working
```
await this.milvusClient.search({
 collection_name: 'my_collection',
 limit: 30,
 output_fields: ['id', 'text'],
 filter: `TEXT_MATCH(text, "episode 281")`,  // Exact match filter
 anns_field: 'sparse_vector',
 data: 'episode 281',  // BM25 ranking query
})
```
For multi-term exact matches:
```
filter: `TEXT_MATCH(text, "foo") and TEXT_MATCH(text, "bar")`
```
## The Mistakes We Made (So You Don’t Have To)

### Dynamic Fields: Critical for Production Flexibility

Initially, we didn’t enable dynamic fields, which was problematic. Schema modifications required dropping and recreating collections in production environments.

```
await this.milvusClient.createCollection({
 collection_name: collectionName,
 fields: fields,
 enable_dynamic_field: true,  // DO THIS
 // ... rest of config
})
```
### Collection Design: Maintain Clear Separation of Concerns

Our architecture uses dedicated collections per feature domain. This modular approach minimizes the impact of schema changes and improves maintainability.

### Memory Usage: Optimize with MMAP

Sparse indexes require significant memory allocation. For large text datasets, we recommend configuring MMAP to utilize disk storage. This approach requires adequate I/O capacity to maintain performance characteristics.

```
// In your Milvus configuration
use_mmap: true
```
## Production Impact and Results

Following the June 2025 deployment of exact match functionality, we observed measurable improvements in user satisfaction metrics and reduced support volume for search-related issues. Our dual-mode approach enables semantic search for exploratory queries while providing precise matching for specific content retrieval.

The key architectural benefit: maintaining a single database system that supports both search paradigms, reducing operational complexity while expanding functionality.

## What’s Next?

We’re experimenting with **hybrid** **queries combining semantic and exact match in a single search**. Imagine: “Find funny clips from episode 281” where “funny” uses semantic search and “episode 281” uses exact match.

The future of search isn’t picking between semantic AI and exact matching. It’s using **both** intelligently in the same system.


How OpusSearch Built Exact Matching for Enterprise RAG with Milvus

How OpusSearch Built Exact Matching for Enterprise RAG with Milvus BM25


Most AI apps rely on a single model for every request. But that approach quickly runs into limits. Large models are powerful yet expensive, even when they're used for simple queries. Smaller models are cheaper and faster but can’t handle complex reasoning. When traffic surges—say your AI app suddenly goes viral with ten million users overnight—the inefficiency of this one-model-for-all setup becomes painfully apparent. Latency spikes, GPU bills explode, and the model that ran fine yesterday starts gasping for air.

And my friend, _you_, the engineer behind this app, have to fix it—fast.

Imagine deploying multiple models of varying sizes and having your system automatically select the best one for each request. Simple prompts go to smaller models; complex queries route to larger ones. That’s the idea behind the [**vLLM Semantic Router**](https://github.com/vllm-project/semantic-router)—a routing mechanism that directs requests based on meaning, not endpoints. It analyzes the semantic content, complexity, and intent of each input to select the most suitable language model, ensuring every query is handled by the model best equipped for it.

To make this even more efficient, the Semantic Router pairs with [**Milvus**](https://milvus.io/), an open-source vector database that serves as a **semantic cache layer**. Before recomputing a response, it checks whether a semantically similar query has already been processed and instantly retrieves the cached result if found. The result: faster responses, lower costs, and a retrieval system that scales intelligently rather than wastefully.

In this post, we’ll dive deeper into how the **vLLM Semantic Router** works, how **Milvus** powers its caching layer, and how this architecture can be applied in real-world AI applications.

## What is a Semantic Router? 

At its core, a **Semantic Router** is a system that decides _which model_ should handle a given request based on its meaning, complexity, and intent. Instead of routing everything to one model, it distributes requests intelligently across multiple models to balance accuracy, latency, and cost.

Architecturally, it’s built on three key layers: **Semantic Routing**, **Mixture of Models (MoM)**, and a **Cache Layer**.

### Semantic Routing Layer 

The **semantic routing layer** is the brain of the system. It analyzes each input—what it’s asking, how complex it is, and what kind of reasoning it requires—to select the model best suited for the job. For example, a simple fact lookup might go to a lightweight model, while a multi-step reasoning query is routed to a larger one. This dynamic routing keeps the system responsive even as traffic and query diversity increase.

![](https://assets.zilliz.com/modern_approach_714403b61c.png)

### The Mixture of Models (MoM) Layer 

The second layer, the **Mixture of Models (MoM)**, integrates multiple models of different sizes and capabilities into one unified system. It’s inspired by the [**Mixture of Experts**](https://zilliz.com/learn/what-is-mixture-of-experts) **(MoE)** architecture, but instead of picking “experts” inside a single large model, it operates across multiple independent models. This design reduces latency, lowers costs, and avoids being locked into any single model provider.

![](https://assets.zilliz.com/MOM_0a3eb61985.png)

### The Cache Layer: Where Milvus Makes the Difference

Finally, the **cache layer**—powered by [Milvus Vector Database](https://milvus.io/)—acts as the system’s memory. Before running a new query, it checks whether a semantically similar request has been processed before. If so, it retrieves the cached result instantly, saving compute time and improving throughput.

Traditional caching systems rely on in-memory key-value stores, matching requests by exact strings or templates. That works fine when queries are repetitive and predictable. But real users rarely type the same thing twice. Once the phrasing changes—even slightly—the cache fails to recognize it as the same intent. Over time, the cache hit rate drops, and performance gains vanish as language naturally drifts.

![](https://assets.zilliz.com/semantic_caching_for_vllm_routing_df889058c9.webp)

To fix this, we need caching that understands _meaning_, not just matching words. That’s where **semantic retrieval** comes in. Instead of comparing strings, it compares embeddings—high-dimensional vector representations that capture semantic similarity. The challenge, though, is scale. Running a brute-force search across millions or billions of vectors on a single machine (with time complexity O(N·d)) is computationally prohibitive. Memory costs explode, horizontal scalability collapses, and the system struggles to handle sudden traffic spikes or long-tail queries.

![](https://assets.zilliz.com/milvus_routing_system_5837b93074.png)

As a distributed vector database purpose-built for large-scale semantic search, **Milvus** brings the horizontal scalability and fault tolerance this cache layer needs. It stores embeddings efficiently across nodes and performs [Approximate Nearest Neighbo](https://zilliz.com/blog/ANN-machine-learning)r (ANN) searches with minimal latency, even at massive scale. With the right similarity thresholds and fallback strategies, Milvus ensures stable, predictable performance—turning the cache layer into a resilient semantic memory for your entire routing system.


## How Developers Are Using Semantic Router + Milvus in Production

The combination of **vLLM Semantic Router** and **Milvus** shines in real-world production environments where speed, cost, and reusability all matter. 

Three common scenarios stand out:


### 1. Customer Service Q\&A

Customer-facing bots handle massive volumes of repetitive queries every day—password resets, account updates, delivery statuses. This domain is both cost- and latency-sensitive, making it ideal for semantic routing. The router sends routine questions to smaller, faster models and escalates complex or ambiguous ones to larger models for deeper reasoning. Meanwhile, Milvus caches previous Q\&A pairs, so when similar queries appear, the system can instantly reuse past answers instead of regenerating them.


### 2. Code Assistance

In developer tools or IDE assistants, many queries overlap—syntax help, API lookups, small debugging hints. By analyzing the semantic structure of each prompt, the router dynamically selects an appropriate model size: lightweight for simple tasks, more capable for multi-step reasoning. Milvus boosts responsiveness further by caching similar coding problems and their solutions, turning prior user interactions into a reusable knowledge base.


### 3. Enterprise Knowledge Base

Enterprise queries tend to repeat over time—policy lookups, compliance references, product FAQs. With Milvus as the semantic cache layer, frequently asked questions and their answers can be stored and retrieved efficiently. This minimizes redundant computation while keeping responses consistent across departments and regions.

Under the hood, the **Semantic Router + Milvus** pipeline is implemented in **Go** and **Rust** for high performance and low latency. Integrated at the gateway layer, it continuously monitors key metrics—like hit rates, routing latency, and model performance—to fine-tune routing strategies in real time.

## How to Quickly Test the Semantic Caching in the Semantic Router

Before deploying semantic caching at scale, it’s useful to validate how it behaves in a controlled setup. In this section, we’ll walk through a quick local test that shows how the Semantic Router uses **Milvus** as its semantic cache. You’ll see how similar queries hit the cache instantly while new or distinct ones trigger model generation—proving the caching logic in action.

### Prerequisites

- Container Environment: Docker + Docker Compose
- Vector Database: Milvus Service
- LLM + Embedding: Project downloaded locally

### 1.Deploy the Milvus Vector Database

Download the deployment files

```
wget https://github.com/Milvus-io/Milvus/releases/download/v2.5.12/Milvus-standalone-docker-compose.yml -O docker-compose.yml
```

Start the Milvus service.

```
docker-compose up -d
```

```
docker-compose ps -a
```

![](https://assets.zilliz.com/Start_the_Milvus_service_211f8b11f1.webp)


### 2. Clone the project

```
git clone https://github.com/vllm-project/semantic-router.git
```

### 3. Download local models

```
cd semantic-router
make download-models
```

![](https://assets.zilliz.com/Download_local_models_6243011fa5.webp)

### 4. Configuration Modifications
Note: Modify the semantic_cache type to milvus

```
vim config.yaml
```

```
semantic_cache:
  enabled: true
  backend_type: "milvus"  # Options: "memory" or "milvus"
  backend_config_path: "config/cache/milvus.yaml"
  similarity_threshold: 0.8
  max_entries: 1000  # Only applies to memory backend
  ttl_seconds: 3600
  eviction_policy: "fifo"
```
Modify the Mmilvus configuration
Note: Fill in the Milvusmilvus service just deployed
```
vim milvus.yaml
```

```
# Milvus connection settings
connection:
  # Milvus server host (change for production deployment)
  host: "192.168.7.xxx"  # For production: use your Milvus cluster endpoint
  # Milvus server port
  port: 19530  # Standard Milvus port
  # Database name (optional, defaults to "default")
  database: "default"
  # Connection timeout in seconds
  timeout: 30
  # Authentication (enable for production)
  auth:
    enabled: false  # Set to true for production
    username: ""    # Your Milvus username
    password: ""    # Your Milvus password
  # TLS/SSL configuration (recommended for production)
  tls:
    enabled: false      # Set to true for secure connections
    cert_file: ""       # Path to client certificate
    key_file: ""        # Path to client private key
    ca_file: ""         # Path to CA certificate
# Collection settings
collection:
  # Name of the collection to store cache entries
  name: "semantic_cache"
  # Description of the collection
  description: "Semantic cache for LLM request-response pairs"
  # Vector field configuration
  vector_field:
    # Name of the vector field
    name: "embedding"
    # Dimension of the embeddings (auto-detected from model at runtime)
    dimension: 384  # This value is ignored - dimension is auto-detected from the embedding model
    # Metric type for similarity calculation
    metric_type: "IP"  # Inner Product (cosine similarity for normalized vectors)
  # Index configuration for the vector field
  index:
    # Index type (HNSW is recommended for most use cases)
    type: "HNSW"
    # Index parameters
    params:
      M: 16              # Number of bi-directional links for each node
      efConstruction: 64  # Search scope during index construction
```

### 5. Start the project
Note: It is recommended to modify some Dockerfile dependencies to domestic sources

```
docker compose --profile testing up --build
```

![](https://assets.zilliz.com/Start_the_project_4e7c2a8332.webp)

### 6. Test Requests
Note: Two requests in total (no cache and cache hit)
First request:
```
echo "=== 第一次请求（无缓存状态） ===" && \
time curl -X POST http://localhost:8801/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer test-token" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What are the main renewable energy sources?"}
    ],
    "temperature": 0.7
  }' | jq .
```
Output:
```
real    0m16.546s
user    0m0.116s
sys     0m0.033s
```
Second request:
```
echo "=== 第二次请求（缓存状态） ===" && \
time curl -X POST http://localhost:8801/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer test-token" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What are the main renewable energy sources?"}
    ],
    "temperature": 0.7
  }' | jq .
```
Output:
```
real    0m2.393s
user    0m0.116s
sys     0m0.021s
```
This test demonstrates Semantic Router's semantic caching in action. By leveraging Milvus as the vector database, it efficiently matches semantically similar queries, improving response times when users ask the same or similar questions.

## Conclusion

As AI workloads grow and cost optimization becomes essential, the combination of vLLM Semantic Router and [Milvus](https://milvus.io/) provides a practical way to scale intelligently. By routing each query to the right model and caching semantically similar results with a distributed vector database, this setup cuts compute overhead while keeping responses fast and consistent across use cases.

In short, you get smarter scaling—less brute force, more brains.

If you’d like to explore this further, join the conversation in our [Milvus Discord](https://discord.com/invite/8uyFbECzPX) or open an issue on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute[ Milvus Office Hours session](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) for one-on-one guidance, insights, and technical deep dives from the team behind Milvus.


Scale Your AI Apps the Smart Way with vLLM Semantic Router and Milvus

vLLM Semantic Router + Milvus: How Semantic Routing and Caching Build Scalable AI Systems the Smart Way


A Deep Dive into Milvus Analyzer for Full-Text Search


Tokenize Smarter, Retrieve Better: A Deep Dive into Milvus Analyzer for Full-Text Search



Building a robust RAG system usually starts with **document** [**chunking**](https://zilliz.com/learn/guide-to-chunking-strategies-for-rag#Chunking)—splitting large texts into manageable pieces for embedding and retrieval. Common strategies include:

*   **Fixed‑size chunks** (e.g., every 512 tokens)
    
*   **Variable‑size chunks** (e.g., paragraph or sentence boundaries)
    
*   **Sliding windows** (overlapping spans)
    
*   **Recursive chunking** (hierarchical splits)
    
*   **Semantic chunking** (grouping by topic)
    

While these methods have their merits, they often fracture long‑range context. To address this challenge, Jina AI creates a Late Chunking approach: embed the entire document first, then carve out your chunks. 

In this article, we'll explore how Late Chunking works and demonstrate how combining it with [Milvus](https://milvus.io/)—a high-performance open-source vector database built for similarity search—can dramatically improve your RAG pipelines. Whether you're building enterprise knowledge bases, AI-driven customer support, or advanced search applications, this walkthrough will show you how to manage embeddings more effectively at scale.

## What Is Late Chunking?

Traditional chunking methods can break important connections when key information spans multiple chunks—resulting in poor retrieval performance. 

Consider these release notes for Milvus 2.4.13, split into two chunks like below: 

![](https://assets.zilliz.com/Figure1_Chunking_Milvus2_4_13_Release_Note_fe7fbdb833.png)


_Figure 1. Chunking Milvus 2.4.13 Release Note_


If you query, “What are the new features in Milvus 2.4.13?”, a standard embedding model may fail to link “Milvus 2.4.13” (in Chunk 1) with its features (in Chunk 2). The result? Weaker vectors and lower retrieval accuracy.

Heuristic fixes—such as sliding windows, overlapping contexts, and repeated scans—provide partial relief but no guarantees.

**Traditional chunking** follows this pipeline:

1.  **Pre‑chunk** text (by sentences, paragraphs, or max token length).
    
2.  **Embed** each chunk separately.
    
3.  **Aggregate** token embeddings (e.g., via average pooling) into a single chunk vector.
    

**Late Chunking** flips the pipeline:

1.  **Embed first**: Run a long‑context transformer over the full document, generating rich token embeddings that capture global context.
    
2.  **Chunk later**: Average‑pool contiguous spans of those token embeddings to form your final chunk vectors.
    
![](https://assets.zilliz.com/Figure2_Naive_Chunkingvs_Late_Chunking_a94d30b6ea.png)


_Figure 2. Naive Chunking vs. Late Chunking (_[_Source_](https://jina.ai/news/late-chunking-in-long-context-embedding-models/)_)_



By preserving full‑document context in every chunk, Late Chunking yields:

*   **Higher retrieval accuracy**—each chunk is contextually aware.
    
*   **Fewer chunks**—you send more focused text to your LLM, cutting costs and latency.
    

Many long‑context models like jina-embeddings-v2-base-en can process up to 8,192 tokens—equivalent to about a 20-minute read (roughly 5,000 words)—making Late Chunking practical for most real‑world documents.

Now that we understand the “what” and “why” behind Late Chunking, let’s dive into the “how”. In the next section, we’ll guide you through a hands‑on implementation of the Late Chunking pipeline, benchmark its performance against traditional chunking, and validate its real‑world impact using Milvus. This practical walkthrough will bridge theory and practice, showing exactly how to integrate Late Chunking into your RAG workflows.

## Testing Late Chunking

### Basic Implementation

Below are the core functions for Late Chunking. We’ve added clear docstrings to guide you through each step. The function `sentence_chunker` splits the original document into paragraph‑based chunks, returning both the chunk contents and the chunk annotation information `span_annotations` (i.e., the start and end indices of each chunk).

```
def sentence_chunker(document, batch_size=10000):
    nlp = spacy.blank("en")
    nlp.add_pipe("sentencizer", config={"punct_chars": None})
    doc = nlp(document)

    docs = []
    for i in range(0, len(document), batch_size):
        batch = document[i : i + batch_size]
        docs.append(nlp(batch))

    doc = Doc.from_docs(docs)

    span_annotations = []
    chunks = []
    for i, sent in enumerate(doc.sents):
        span_annotations.append((sent.start, sent.end))
        chunks.append(sent.text)

    return chunks, span_annotations
```


The function `document_to_token_embeddings` uses the jinaai/jina-embeddings-v2-base-en model and its tokenizer to produce embeddings for the entire document.

```
def document_to_token_embeddings(model, tokenizer, document, batch_size=4096):
    tokenized_document = tokenizer(document, return_tensors="pt")
    tokens = tokenized_document.tokens()

    outputs = []
    for i in range(0, len(tokens), batch_size):
        
        start = i
        end   = min(i + batch_size, len(tokens))

        batch_inputs = {k: v[:, start:end] for k, v in tokenized_document.items()}

        with torch.no_grad():
            model_output = model(**batch_inputs)

        outputs.append(model_output.last_hidden_state)

    model_output = torch.cat(outputs, dim=1)
    return model_output
```


The function `late_chunking` takes the document’s token embeddings and the original chunk annotation information `span_annotations`, and then produces the final chunk embeddings.

```
def late_chunking(token_embeddings, span_annotation, max_length=None):
    outputs = []
    for embeddings, annotations in zip(token_embeddings, span_annotation):
        if (
            max_length is not None
        ):
            annotations = [
                (start, min(end, max_length - 1))
                for (start, end) in annotations
                if start < (max_length - 1)
            ]
        pooled_embeddings = []
        for start, end in annotations:
            if (end - start) >= 1:
                pooled_embeddings.append(
                    embeddings[start:end].sum(dim=0) / (end - start)
                )
                    
        pooled_embeddings = [
            embedding.detach().cpu().numpy() for embedding in pooled_embeddings
        ]
        outputs.append(pooled_embeddings)

    return outputs
```



For example, chunking with jinaai/jina-embeddings-v2-base-en:

```
tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True)
model     = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True)

# First chunk the text as normal, to obtain the beginning and end points of the chunks.
chunks, span_annotations = sentence_chunker(document)
# Then embed the full document.
token_embeddings = document_to_token_embeddings(model, tokenizer, document)
# Then perform the late chunking
chunk_embeddings = late_chunking(token_embeddings, [span_annotations])[0]
```


_Tip:_ Wrapping your pipeline in functions makes it easy to swap in other long‑context models or chunking strategies.

### Comparison with Traditional Embedding Methods

To further demonstrate the advantages of Late Chunking, we also compared it to traditional embedding approaches, using a set of sample documents and queries.

And let’s revisit our Milvus 2.4.13 release note example:

```
Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection. This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery. Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency. We highly recommend upgrading to this release for better performance and stability.
```


We measure [cosine similarity](https://zilliz.com/blog/similarity-metrics-for-vector-search#Cosine-Similarity) between the query embedding ("milvus 2.4.13") and each chunk:

```
cos_sim = lambda x, y: np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))

milvus_embedding = model.encode('milvus 2.4.13')

for chunk, late_chunking_embedding, traditional_embedding in zip(chunks, chunk_embeddings, embeddings_traditional_chunking):
    print(f'similarity_late_chunking("milvus 2.4.13", "{chunk}")')
    print('late_chunking: ', cos_sim(milvus_embedding, late_chunking_embedding))
    print(f'similarity_traditional("milvus 2.4.13", "{chunk}")')
    print('traditional_chunking: ', cos_sim(milvus_embedding, traditional_embeddings))
```


Late Chunking consistently outperformed traditional chunking, yielding higher cosine similarities across every chunk. This confirms that embedding the full document first preserves global context more effectively.

```
similarity_late_chunking("milvus 2.4.13", "Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection.")
late_chunking: 0.8785206
similarity_traditional("milvus 2.4.13", "Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection.")
traditional_chunking: 0.8354263

similarity_late_chunking("milvus 2.4.13", "This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery.")
late_chunking: 0.84828955
similarity_traditional("milvus 2.4.13", "This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery.")
traditional_chunking: 0.7222632

similarity_late_chunking("milvus 2.4.13", "Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency.")
late_chunking: 0.84942204
similarity_traditional("milvus 2.4.13", "Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency.")
traditional_chunking: 0.6907381

similarity_late_chunking("milvus 2.4.13", "We highly recommend upgrading to this release for better performance and stability.")
late_chunking: 0.85431844
similarity_traditional("milvus 2.4.13", "We highly recommend upgrading to this release for better performance and stability.")
traditional_chunking: 0.71859795
```


We can see that embedding the full paragraph first ensures each chunk carries the “`Milvus 2.4.13`” context—boosting similarity scores and retrieval quality.

### **Testing Late Chunking in Milvus**

Once chunk embeddings are generated, we can store them in Milvus and perform queries. The following code inserts chunk vectors into the collection.

#### **Importing Embeddings into Milvus**

```
batch_data=[]
for i in range(len(chunks)):
    data = {
            "content": chunks[i],
            "embedding": chunk_embeddings[i].tolist(),
        }

    batch_data.append(data)

res = client.insert(
    collection_name=collection,
    data=batch_data,
)
```



#### Querying and Validation

To validate the accuracy of Milvus queries, we compare its retrieval results to brute-force cosine similarity scores calculated manually. If both methods return consistent top‑k results, we can be confident that Milvus’s search accuracy is reliable.

We compare Milvus’s native search with a brute‑force cosine similarity scan:

```
def late_chunking_query_by_milvus(query, top_k = 3):
    query_vector = model(**tokenizer(query, return_tensors="pt")).last_hidden_state.mean(1).detach().cpu().numpy().flatten()

    res = client.search(
                collection_name=collection,
                data=[query_vector.tolist()],
                limit=top_k,
                output_fields=["id", "content"],
            )

    return [item.get("entity").get("content") for items in res for item in items]

def late_chunking_query_by_cosine_sim(query, k = 3):
    cos_sim = lambda x, y: np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))
    query_vector = model(**tokenizer(query, return_tensors="pt")).last_hidden_state.mean(1).detach().cpu().numpy().flatten()

    results = np.empty(len(chunk_embeddings))
    for i, (chunk, embedding) in enumerate(zip(chunks, chunk_embeddings)):
        results[i] = cos_sim(query_vector, embedding)

    results_order = results.argsort()[::-1]
    return np.array(chunks)[results_order].tolist()[:k]
```


This confirms Milvus returns the same top‑k chunks as a manual cosine‑sim scan.

```
> late_chunking_query_by_milvus("What are new features in milvus 2.4.13", 3)

['\n\n### Features\n\n- Dynamic replica adjustment for loaded collections ([#36417](https://github.com/milvus-io/milvus/pull/36417))\n- Sparse vector MMAP in growing segment types ([#36565](https://github.com/milvus-io/milvus/pull/36565))...
```

```
> late_chunking_query_by_cosine_sim("What are new features in milvus 2.4.13", 3)

['\n\n### Features\n\n- Dynamic replica adjustment for loaded collections ([#36417](https://github.com/milvus-io/milvus/pull/36417))\n- Sparse vector MMAP in growing segment types (#36565)...
```


So both methods yield the same top-3 chunks, confirming Milvus’s accuracy.

## Conclusion


In this article, we took a deep dive into the mechanics and benefits of Late Chunking. We began by identifying the shortcomings of traditional chunking approaches, particularly when handling long documents where preserving context is crucial. We introduced the concept of Late Chunking—embedding the entire document before slicing it into meaningful chunks—and showed how this preserves global context, leading to improved semantic similarity and retrieval accuracy.

We then walked through a hands-on implementation using Jina AI’s jina-embeddings-v2-base-en model and evaluated its performance compared to traditional methods. Finally, we demonstrated how to integrate the chunk embeddings into Milvus for scalable and accurate vector search.

Late Chunking offers a **context‑first** approach to embedding—perfect for long, complex documents where context matters most. By embedding the entire text upfront and slicing later, you gain:

*   🔍 **Sharper retrieval accuracy**
    
*   ⚡ **Lean, focused LLM prompts**
    
*   🛠️ **Simple integration** with any long‑context model


Smarter Retrieval for RAG: Late Chunking with Jina Embeddings v2 and Milvus


How to Choose the Right Embedding Model for RAG


From Word2Vec to LLM2Vec: How to Choose the Right Embedding Model for RAG



Performance is at the heart of Milvus. Under normal conditions, a search request within Milvus completes in just milliseconds. But what happens when your cluster slows down—when search latency stretches into whole seconds instead? 

Slow searches don’t happen often, but they can surface at scale or under complex workloads. And when they do, they matter: they disrupt user experience, skew application performance, and often expose hidden inefficiencies in your setup.

In this post, we’ll walk through how to triage slow requests in Milvus and share practical steps you can take to keep latency predictable, stable, and consistently low.

## Identifying Slow Searches

Diagnosing a slow request starts with two questions: **how often does it happen, and where is the time going?** Milvus gives you both answers through metrics and logs.

### Milvus Metrics

Milvus exports detailed metrics you can monitor in Grafana dashboards. 

![](https://assets.zilliz.com/image_2_64a5881bf2.png)

![](https://assets.zilliz.com/image_3_b7b8b369ec.png)

Key panels include:

*   **Service Quality → Slow Query**: Flags any request exceeding proxy.slowQuerySpanInSeconds (default: 5s). These are also marked in Prometheus.
    
*   **Service Quality → Search Latency**: Shows overall latency distribution. If this looks normal, but end users still see delays, the problem is likely outside Milvus—in the network or application layer.
    
*   **Query Node → Search Latency by Phase**: Breaks latency into queue, query, and reduce stages. For deeper attribution, panels such as _Scalar_ _Filter Latency_, _Vector Search Latency_, and _Wait tSafe Latency_ reveal which stage dominates.
    

### Milvus Logs

Milvus also logs any request lasting more than one second, tagged with markers like [Search slow]. These logs show _which_ queries are slow, complementing the _where_ insights from metrics. As a rule of thumb:

*   **< 30 ms** → healthy search latency in most scenarios
    
*   **> 100 ms** → worth investigating
    
*   **> 1 s** → definitely slow and requires attention
    

Example log:

```
[2025/08/23 19:22:19.900 +00:00] [INFO] [proxy/impl.go:3141] ["Search slow"] [traceID=9100b3092108604716f1472e4c7d54e4] [role=proxy] [db=default] [collection=my_repos] [partitions="[]"] [dsl="user == \"milvus-io\" && repo == \"proxy.slowQuerySpanInSeconds\""] [len(PlaceholderGroup)=8204] [OutputFields="[user,repo,path,descripion]"] [search_params="[{\"key\":\"topk\",\"value\":\"10\"},{\"key\":\"metric_type\",\"value\":\"COSINE\"},{\"key\":\"anns_field\",\"value\":\"vector\"},{\"key\":\"params\",\"value\":\"{\\\"nprobe\\\":256,\\\"metric_type\\\":\\\"COSINE\\\"}\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [guarantee_timestamp=460318735832711168] [nq=1] [duration=5m12.002784545s] [durationPerNq=5m12.002784545s]
```

In short, **metrics tell you where the time is going; logs tell you which queries are hit.**

## Analyzing Root Cause

### Heavy Workload

A common cause of slow requests is an excessive workload. When a request has a very large **NQ** (number of queries per request), it can run for an extended period and monopolize query node resources. Other requests stack up behind it, resulting in rising queue latency. Even if each request has a small NQ, a very high overall throughput (QPS) can still cause the same effect, as Milvus may merge concurrent search requests internally.

![ ](https://assets.zilliz.com/high_workload_cf9c75e24c.png)

**Signals to watch for:**

*   All queries show unexpectedly high latency.
    
*   Query Node metrics report high **in-queue latency**.
    
*   Logs show a request with a large NQ and a long total duration, but a relatively small durationPerNQ—indicating that one oversized request is dominating resources.
    

**How to fix it:**

*   **Batch queries**: Keep NQ modest to avoid overloading a single request.
    
*   **Scale out query nodes**: If high concurrency is a regular part of your workload, add query nodes to spread the load and maintain low latency.
    

### Inefficient Filtering

Another common bottleneck comes from inefficient filters. If filter expressions are poorly conducted or fields lack scalar indexes, Milvus may fall back to a **full scan** instead of scanning a small, targeted subset. JSON filters and strict consistency settings can further increase overhead.

![ ](https://assets.zilliz.com/inefficient_filtering_e524615d63.png)

**Signals to watch for:**

*   High **Scalar Filter Latency** in Query Node metrics.
    
*   Noticeable latency spikes only when filters are applied.
    
*   Long **Wait tSafe Latency** if strict consistency is enabled.
    

**How to fix it:**

*   **Simplify filter expressions**: Reduce query plan complexity by optimizing filters. For example, replace long OR chains with an IN expression:
    

```
# Replace chains of OR conditions with IN
tag = {"tag": ["A", "B", "C", "D"]}
filter_expr = "tag IN {tag}"
```

*   Milvus also introduces a filter expression templating mechanism designed to improve efficiency by reducing the time spent parsing complex expressions. See [this document](https://milvus.io/docs/filtering-templating.md) for more details.

 *   **Add proper indexes**: Avoid full scans by creating scalar indexes on fields used in filters.
     
 *   **Handle JSON efficiently**: Milvus 2.6 introduced path and flat indexes for JSON fields, enabling efficient handling of JSON data. JSON shredding is also on [the roadmap](https://milvus.io/docs/roadmap.md) to further improve performance. Refer to [the JSON field document](https://milvus.io/docs/use-json-fields.md#JSON-Field) for additional information.  
     
 *   **Tune consistency level**: Use *Bounded* or *Eventually* consistent reads when strict guarantees are not required, reducing *tSafe* wait time.
    

### Improper Choice of Vector Index

[Vector indexes](https://milvus.io/docs/index-explained.md) are not one-size-fits-all. Selecting the wrong index can significantly impact latency. In-memory indexes deliver the fastest performance but consume more memory, while on-disk indexes save memory at the cost of speed. Binary vectors also require specialized indexing strategies.

![ ](https://assets.zilliz.com/image_4_25fa1b9c13.png)

**Signals to watch for:**

*   High Vector Search Latency in Query Node metrics.
    
*   Disk I/O saturation when using DiskANN or MMAP.
    
*   Slower queries immediately after restart due to cache cold start.
    

**How to fix it:**

*   **Match index to workload (float vectors):**
    
    *   **HNSW** — best for in-memory use cases with high recall and low latency.
        
    *   **IVF family** — flexible trade-offs between recall and speed.
        
    *   **DiskANN** — supports billion-scale datasets, but requires strong disk bandwidth.
        
*   **For binary vectors:** Use the [MINHASH_LSH index](https://milvus.io/docs/minhash-lsh.md) (introduced in Milvus 2.6) with the MHJACCARD metric to efficiently approximate Jaccard similarity.
    
*   **Enable** [**MMAP**](https://milvus.io/docs/mmap.md): Map index files into memory instead of keeping them fully resident to strike a balance between latency and memory usage.
    
*   **Tune index/search parameters**: Adjust settings to balance recall and latency for your workload.
    
*   **Mitigate cold starts**: Warm up frequently accessed segments after a restart to avoid initial query slowness.
    

### Runtime & Environment Conditions

Not all slow queries are caused by the query itself. Query nodes often share resources with background jobs, such as compaction, data migration, or index building. Frequent upserts can generate many small, unindexed segments, forcing searches to scan raw data. In some cases, version-specific inefficiencies can also introduce latency until patched.

![ ](https://assets.zilliz.com/img_v3_02q5_4dd2e545_93dc_4c58_b609_d76d50c2013g_aad0a89208.png)

**Signals to watch for:**

*   CPU usage spikes during background jobs (compaction, migration, index builds).
    
*   Disk I/O saturation affecting query performance.
    
*   Very slow cache warm-up after a restart.
    
*   Large numbers of small, unindexed segments (from frequent upserts).
    
*   Latency regressions tied to specific Milvus versions.
    

**How to fix it:**

*   **Reschedule background tasks** (e.g., compaction) to off-peak hours.
    
*   **Release unused collections** to free memory.
    
*   **Account for warm-up time** after restarts; pre-warm caches if needed.
    
*   **Batch upserts** to reduce the creation of tiny segments and let compaction keep up.
    
*   **Stay current**: upgrade to newer Milvus versions for bug fixes and optimizations.
    
*   **Provision resources**: dedicate extra CPU/memory to latency-sensitive workloads.
    

By matching each signal with the right action, most slow queries can be resolved quickly and predictably.

## Best Practices to Prevent Slow Searches

The best debugging session is the one you never need to run. In our experience, a few simple habits go a long way toward preventing slow queries in Milvus:

*   **Plan resource allocation** to avoid CPU and disk contention.
    
*   **Set proactive alerts** for both failures and latency spikes.
    
*   **Keep filter expressions** short, simple, and efficient.
    
*   **Batch upserts** and keep NQ/QPS at sustainable levels.
    
*   **Index all fields** that are used in filters.
    

Slow queries in Milvus are rare, and when they do appear, they usually have clear, diagnosable causes. With metrics, logs, and a structured approach, you can quickly identify and resolve issues. This is the same playbook our support team uses every day — and now it’s yours too.

We hope this guide provides not only a troubleshooting framework but also the confidence to keep your Milvus workloads running smoothly and efficiently.

## 💡 Want to dive deeper?

*   Join the [**Milvus Discord**](https://discord.com/invite/8uyFbECzPX) to ask questions, share experiences, and learn from the community.
    
*   Sign up for our [**Milvus Office Hours**](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) to speak directly with the team and receive hands-on assistance with your workloads.


How to Debug Slow Search Requests in Milvus



For a long time, many of us — myself included — treated LLM hallucinations as nothing more than defects. An entire toolchain has been built around eliminating them: retrieval systems, guardrails, fine-tuning, and more. These safeguards are still valuable. But the more I’ve studied how models actually generate responses — and how systems like [**Milvus**](https://milvus.io/) fit into broader AI pipelines — the less I believe hallucinations are simply failures. In fact, they can also be the spark of AI creativity.

If we look at human creativity, we find the same pattern. Every breakthrough relies on imaginative leaps. But those leaps never come out of nowhere. Poets first master rhythm and meter before they break the rules. Scientists rely on established theories before venturing into untested territory. Progress depends on these leaps, as long as they are grounded in solid knowledge and understanding.

LLMs operate in much the same way. Their so-called “hallucinations” or “leaps”— analogies, associations, and extrapolations — emerge from the same generative process that allows models to make connections, extend knowledge, and surface ideas beyond what they’ve been explicitly trained on. Not every leap succeeds, but when it does, the results can be compelling.

That’s why I see **Context Engineering** as the critical next step. Rather than trying to eliminate every hallucination, we should focus on _steering_ them. By designing the right context, we can strike a balance — keeping models imaginative enough to explore new ground, while ensuring they remain anchored enough to be trusted.

## What is Context Engineering?


So what exactly do we mean by _context engineering_? The term may be new, but the practice has been evolving for years. Techniques such as RAG, prompting, function calling, and MCP are all early attempts at solving the same problem: providing models with the right environment to produce useful results. Context engineering is about unifying those approaches into a coherent framework.

## The Three Pillars of Context Engineering


Effective context engineering rests on three interconnected layers:

![](https://assets.zilliz.com/context_engineering_1_8f2b39c5e7.png)

### 1\. The Instructions Layer — Defining Direction

This layer includes prompts, few-shot examples, and demonstrations. It’s the model’s navigation system: not just a vague “go north,” but a clear route with waypoints. Well-structured instructions set boundaries, define goals, and reduce ambiguity in model behavior.

### 2\. The Knowledge Layer — Supplying Ground Truth

Here we place the facts, code, documents, and state that the model needs to reason effectively. Without this layer, the system improvises from incomplete memory. With it, the model can ground its outputs in domain-specific data. The more accurate and relevant the knowledge, the more reliable the reasoning.

### 3\. The Tools Layer — Enabling Action and Feedback

This layer covers APIs, function calls, and external integrations. It’s what enables the system to move beyond reasoning to execution—retrieving data, performing calculations, or triggering workflows. Just as importantly, these tools provide real-time feedback that can be looped back into the model’s reasoning. That feedback is what enables correction, adaptation, and continuous improvement. In practice, this is what transforms LLMs from passive responders into active participants in a system.

These layers aren’t silos—they reinforce each other. Instructions set the destination, knowledge provides the information to work with, and tools turn decisions into action and feed results back into the loop. Orchestrated well, they create an environment where models can be both creative and dependable.

## The Long Context Challenges: When More Becomes Less

Many AI models now advertise million-token windows—enough for ~75,000 lines of code or a 750,000-word document. But more context doesn’t automatically yield better results. In practice, very long contexts introduce distinct failure modes that can degrade reasoning and reliability.

### Context Poisoning — When Bad Information Spreads

Once false information enters the working context—whether in goals, summaries, or intermediate state—it can derail the entire reasoning process. [DeepMind’s Gemini 2.5 report](https://arxiv.org/pdf/2507.06261) provides a clear example. An LLM agent playing Pokémon misread the game state and decided its mission was to “catch the uncatchable legendary.” That incorrect goal was recorded as fact, leading the agent to generate elaborate but impossible strategies. 

As shown in the excerpt below, the poisoned context trapped the model in a loop—repeating errors, ignoring common sense, and reinforcing the same mistake until the entire reasoning process collapsed.

![](https://assets.zilliz.com/Figure_1_Excerpt_from_Gemini_2_5_Tech_Paper_e89adf9eed.png)

Figure 1: Excerpt from [Gemini 2.5 Tech Paper](https://arxiv.org/pdf/2507.06261)

### Context Distraction — Lost in the Details

As context windows expand, models can start to overweight the transcript and underuse what they learned during training. DeepMind’s Gemini 2.5 Pro, for example, supports a million-token window but [begins to drift around ~100,000 tokens](https://arxiv.org/pdf/2507.06261)—recycling past actions instead of generating new strategies. [Databricks’ research](https://www.databricks.com/blog/long-context-rag-performance-llms) shows that smaller models, like Llama 3.1-405B, reach that limit far sooner at roughly ~32,000 tokens. It’s a familiar human effect: too much background reading, and you lose the plot.

![](https://assets.zilliz.com/Figure_2_Excerpt_from_Gemini_2_5_Tech_Paper_56d775c59d.png)

Figure 2: Excerpt from [Gemini 2.5 Tech Paper](https://arxiv.org/pdf/2507.06261)



![](https://assets.zilliz.com/Figure_3_Long_context_performance_of_GPT_Claude_Llama_Mistral_and_DBRX_models_on_4_curated_RAG_datasets_Databricks_Docs_QA_Finance_Bench_Hot_Pot_QA_and_Natural_Questions_Source_Databricks_99086246b9.png)

_Figure 3: Long context performance of GPT, Claude, Llama, Mistral and DBRX models on 4 curated RAG datasets (Databricks DocsQA, FinanceBench, HotPotQA and Natural Questions) \[Source:_ [_Databricks_](https://www.databricks.com/blog/long-context-rag-performance-llms)_\]_ 

### Context Confusion — Too Many Tools in the Kitchen

Adding more tools doesn’t always help. The [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) shows that when the context displays extensive tool menus—often with many irrelevant options—model reliability decreases, and tools are invoked even when none are needed. One clear example: a quantized Llama 3.1-8B failed with 46 tools available, but succeeded when the set was reduced to 19. It’s the paradox of choice for AI systems—too many options, worse decisions.

### Context Clash — When Information Conflicts

Multi-turn interactions add a distinct failure mode: early misunderstandings compound as the dialogue branches. In [Microsoft and Salesforce experiments](https://arxiv.org/pdf/2505.06120v1), both open- and closed-weight LLMs performed markedly worse in multi-turn vs. single-turn settings—an average 39% drop across six generation tasks. Once a wrong assumption enters the conversation state, subsequent turns inherit it and amplify the error.

![](https://assets.zilliz.com/Figure_4_LL_Ms_get_lost_in_multi_turn_conversations_in_experiments_21f194b02d.png)

_Figure 4: LLMs get lost in multi-turn conversations in experiments_ 

The effect shows up even in frontier models. When benchmark tasks were distributed across turns, the performance score of OpenAI’s o3 model fell from **98.1** to **64.1**. An initial misread effectively “sets” the world model; each reply builds on it, turning a small contradiction into a hardened blind spot unless explicitly corrected.

![](https://assets.zilliz.com/Figure_4_The_performance_scores_in_LLM_multi_turn_conversation_experiments_414d3a0b3f.png)

_Figure 4: The performance scores in LLM multi-turn conversation experiments_ 

## Six Strategies to Tame Long Context

The answer to long-context challenges isn’t to abandon the capability—it’s to engineer it with discipline. Here are six strategies we’ve seen work in practice:

### Context Isolation 

Break complex workflows into specialized agents with isolated contexts. Each agent focuses on its own domain without interference, reducing the risk of error propagation. This not only improves accuracy but also enables parallel execution, much like a well-structured engineering team.

### Context Pruning 

Regularly audit and trim the context. Remove redundant details, stale information, and irrelevant traces. Think of it as refactoring: clean out dead code and dependencies, leaving only the essentials. Effective pruning requires explicit criteria for what belongs and what doesn’t.

### Context Summarization 

Long histories don’t need to be carried around in full. Instead, condense them into concise summaries that capture only what is essential for the next step. Good summarization retains the critical facts, decisions, and constraints, while eliminating repetition and unnecessary details. It’s like replacing a 200-page spec with a one-page design brief that still gives you everything you need to move forward.

### Context Offloading

Not every detail needs to be part of the live context. Persist non-critical data in external systems—knowledge bases, document stores, or vector databases like Milvus—and fetch it only when needed. This lightens the model’s cognitive load while keeping background information accessible.

### Strategic RAG 

Information retrieval is powerful only if it’s selective. Introduce external knowledge through rigorous filtering and quality controls, ensuring the model consumes relevant and accurate inputs. As with any data pipeline: garbage in, garbage out—but with high-quality retrieval, the context becomes an asset, not a liability.

### Optimized Tool Loading

More tools don’t equal better performance. Studies show reliability drops sharply beyond ~30 available tools. Load only the functions a given task requires, and gate access to the rest. A lean toolbox fosters precision and reduces the noise that can overwhelm decision-making.

## The Infrastructure Challenge of Context Engineering

Context engineering is only as effective as the infrastructure it runs on. And today’s enterprises are hitting a perfect storm of data challenges:

### Scale Explosion — From Terabytes to Petabytes

Today, data growth has redefined the baseline. Workloads that once fit comfortably in a single database now span petabytes, demanding distributed storage and compute. A schema change that used to be a one-line SQL update can cascade into a full orchestration effort across clusters, pipelines, and services. Scaling isn’t simply about adding hardware—it’s about engineering for coordination, resilience, and elasticity at a scale where every assumption gets stress-tested.

### Consumption Revolution — Systems That Speak AI

AI agents don’t just query data; they generate, transform, and consume it continuously at machine speeds. Infrastructure designed just for human-facing applications can’t keep up. To support agents, systems must provide low-latency retrieval, streaming updates, and write-heavy workloads without breaking. In other words, the infrastructure stack must be built to “speak AI” as its native workload, not as an afterthought.

### Multimodal Complexity — Many Data Types, One System

AI workloads blend text, images, audio, video, and high-dimensional embeddings, each with rich metadata attached. Managing this heterogeneity is the crux of practical context engineering. The challenge isn’t just storing diverse objects; it’s indexing them, retrieving them efficiently, and keeping semantic consistency across modalities. A truly AI-ready infrastructure must treat multimodality as a first-class design principle, not a bolt-on feature.

## Milvus + Loon: Purpose-Built Data Infrastructure for AI 

The challenges of scale, consumption, and multimodality can’t be solved with theory alone—they demand infrastructure that is purpose-built for AI. That’s why we at [Zilliz](https://zilliz.com/) designed **Milvus** and **Loon** to work together, addressing both sides of the problem: high-performance retrieval at runtime and large-scale data processing upstream.

*   [**Milvus**](https://milvus.io/): the most widely adopted open-source vector database optimized for high-performance vector retrieval and storage. 
    
*   **Loon**: our upcoming cloud-native multimodal data lake service designed to process and organize massive-scale multimodal data before it ever reaches the database. Stay tuned. 
    
![](https://assets.zilliz.com/loon_milvus_min_76aaa39b4e.png)


### Lightning-Fast Vector Search

**Milvus** is built from the ground up for vector workloads. As the serving layer, it delivers sub-10ms retrieval across hundreds of millions—or even billions—of vectors, whether derived from text, images, audio, or video. For AI applications, retrieval speed isn’t a “nice to have.” It’s what determines whether an agent feels responsive or sluggish, whether a search result feels relevant or out of step. Performance here is directly visible in the end-user experience.

### Multimodal Data Lake Service at Scale

**Loon** is our upcoming multimodal data lake service, designed for massive-scale offline processing and analytics of unstructured data. It complements Milvus on the pipeline side, preparing data before it ever reaches the database. Real-world multimodal datasets—spanning text, images, audio, and video—are often messy, with duplication, noise, and inconsistent formats. Loon takes care of this heavy lifting using distributed frameworks like Ray and Daft, compressing, deduplicating, and clustering the data before streaming it directly into Milvus. The result is simple: no staging bottlenecks, no painful format conversions—just clean, structured data that models can use immediately.

### Cloud-Native Elasticity

Both systems are built cloud-native, with storage and compute scaling independently. That means as workloads grow from gigabytes to petabytes, you can balance resources between real-time serving and offline training, rather than overprovisioning for one or undercutting the other.

### Future-Proof Architecture

Most importantly, this architecture is designed to grow with you. Context engineering is still evolving. Right now, most teams are focused on semantic search and RAG pipelines. But the next wave will demand more—integrating multiple data types, reasoning across them, and powering agent-driven workflows.

With Milvus and Loon, that transition doesn’t require ripping out your foundation. The same stack that supports today’s use cases can extend naturally into tomorrow’s. You add new capabilities without starting over, which means less risk, lower cost, and a smoother path as AI workloads become more complex.

## Your Next Move

Context engineering isn’t just another technical discipline—it’s how we unlock AI’s creative potential while keeping it grounded and reliable. If you’re ready to put these ideas into practice, start where it matters most. 

*   [**Experiment with Milvus**](https://milvus.io/docs/overview.md) to see how vector databases can anchor retrieval in real-world deployments.
    
*   [**Follow Milvus**](https://www.linkedin.com/company/the-milvus-project/) for updates on Loon’s release and insights into managing large-scale multimodal data.
    
*   [**Join the Zilliz community on Discord**](https://discord.com/invite/8uyFbECzPX) to share strategies, compare architectures, and help shape best practices.

The companies that master context engineering today will shape the AI landscape tomorrow. Don’t let infrastructure be the constraint—build the foundation your AI creativity deserves.



If Context Engineering Done Right, Hallucinations Can Spark AI Creativity


When Context Engineering Is Done Right, Hallucinations Can Be the Spark of AI Creativity



AI agents are becoming a core pattern in applied AI. More projects are moving past single prompts and wiring models into decision-making loops. That’s exciting, but it also means managing state, coordinating tools, handling branches, and adding human handoffs—things that aren’t immediately obvious.

[**LangGraph**](https://github.com/langchain-ai/langgraph) is a strong choice for this layer. It is an AI framework that provides loops, conditionals, persistence, human-in-the-loop controls, and streaming—enough structure to turn an idea into a real multi-agent app. However, LangGraph has a steep learning curve. Its documentation moves quickly, the abstractions take time to get used to, and jumping from a simple demo to something that feels like a product can be frustrating. 

Recently, I started using [**langgraph-up-react**](https://github.com/webup/langgraph-up-react)—a ready-to-use LangGraph + ReAct template for ReAct agents. It trims setup, ships with sane defaults, and lets you focus on behavior instead of boilerplate. In this post, I’ll walk through how to get started with LangGraph using this template. 


## Understanding ReAct Agents

Before diving into the template itself, it’s worth looking at the kind of agent we’ll be building. One of the most common patterns today is the **ReAct (Reason + Act)** framework, first introduced in Google’s 2022 paper _“_[_ReAct: Synergizing Reasoning and Acting in Language Models._](https://arxiv.org/abs/2210.03629)_”_

The idea is straightforward: instead of treating reasoning and action as separate, ReAct combines them into a feedback loop that looks a lot like human problem solving. The agent **reasons** about the problem, **acts** by calling a tool or API, and then **observes** the result before deciding what to do next. This simple cycle—reason → act → observe—lets agents adapt dynamically instead of following a fixed script.

Here’s how the pieces fit together:

- **Reason**: The model breaks problems into steps, plans strategies, and can even correct mistakes mid-way.

- **Act**: Based on its reasoning, the agent calls tools—whether that’s a search engine, a calculator, or your own custom API.

- **Observe**: The agent looks at the tool’s output, filters the results, and feeds that back into its next round of reasoning.

This loop has quickly become the backbone of modern AI agents. You’ll see traces of it in ChatGPT plugins, RAG pipelines, intelligent assistants, and even robotics. In our case, it’s the foundation that the `langgraph-up-react` template builds on.


## Understanding LangGraph

Now that we’ve looked at the ReAct pattern, the next question is: how do you actually implement something like that in practice? Out of the box, most language models don’t handle multi-step reasoning very well. Each call is stateless: the model generates an answer and forgets everything as soon as it’s done. That makes it hard to carry intermediate results forward or adjust later steps based on earlier ones.

[**LangGraph**](https://github.com/langchain-ai/langgraph) closes this gap. Instead of treating every prompt as a one-off, it gives you a way to break complex tasks into steps, remember what happened at each point, and decide what to do next based on the current state. In other words, it turns an agent’s reasoning process into something structured and repeatable, rather than a chain of ad-hoc prompts.

You can think of it like a **flowchart for AI reasoning**:

- **Analyze** the user query

- **Select** the right tool for the job

- **Execute** the task by calling the tool

- **Process** the results

- **Check** if the task is complete; if not, loop back and continue reasoning

- **Output** the final answer

Along the way, LangGraph handles **memory storage** so results from earlier steps aren’t lost, and it integrates with an **external tool library** (APIs, databases, search, calculators, file systems, etc.).

That’s why it’s called _LangGraph_: **Lang (Language) + Graph**—a framework for organizing how language models think and act over time.


## Understanding langgraph-up-react

LangGraph is powerful, but it comes with overhead. Setting up state management, designing nodes and edges, handling errors, and wiring in models and tools all take time. Debugging multi-step flows can also be painful—when something breaks, the issue might be in any node or transition. As projects grow, even small changes can ripple through the codebase and slow everything down.

This is where a mature template makes a huge difference. Instead of starting from scratch, a template gives you a proven structure, pre-built tools, and scripts that just work. You skip the boilerplate and focus directly on the agent logic.

[**langgraph-up-react**](https://github.com/webup/langgraph-up-react) is one such template. It’s designed to help you spin up a LangGraph ReAct agent quickly, with:

- 🔧 **Built-in tool ecosystem**: adapters and utilities ready to use out of the box

- ⚡ **Quick start**: simple configuration and a working agent in minutes

- 🧪 **Testing included**: unit tests and integration tests for confidence as you extend

- 📦 **Production-ready setup**: architecture patterns and scripts that save time when deploying

In short, it takes care of the boilerplate so you can focus on building agents that actually solve your business problems.


## Getting Started with the langgraph-up-react Template

Getting the template running is straightforward. Here’s the setup process step by step:

1. Install environment dependencies

```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2. Clone the project

```
git clone https://github.com/webup/langgraph-up-react.git
cd langgraph-up-react
```

3. Install dependencies

```
uv sync --dev
```

4. Configure environment

Copy the example config and add your keys:

```
cp .env.example .env
```

Edit .env and set at least one model provider plus your Tavily API key:

```
TAVILY_API_KEY=your-tavily-api-key      # Required for web search  
DASHSCOPE_API_KEY=your-dashscope-api-key  # Qwen (default recommended)  
OPENAI_API_KEY=your-openai-api-key        # OpenAI or compatible platforms  
# OPENAI_API_BASE=https://your-api-endpoint  # If using OpenAI-compatible API  
REGION=us                # Optional: region flag  
ENABLE_DEEPWIKI=true      # Optional: enable document tools  
```

5. Start the project

```
# Start development server (without UI)
make dev

# Start development server with LangGraph Studio UI
make dev_ui
```

Your dev server will now be up and ready for testing.

![](https://assets.zilliz.com/template_set_up_a42d1819ed.png)

## What Can You Build with langgraph-up-react?

So what can you actually do once the template is up and running? Here are two concrete examples that show how it can be applied in real projects.


### Enterprise Knowledge Base Q&A (Agentic RAG)

A common use case is an internal Q&A assistant for company knowledge. Think product manuals, technical docs, FAQs—information that’s useful but scattered. With `langgraph-up-react`, you can create an agent that indexes these documents in a [**Milvus**](https://milvus.io/) vector database, retrieves the most relevant passages, and generates accurate answers grounded in context.

For deployment, Milvus offers flexible options: **Lite** for quick prototyping, **Standalone** for mid-sized production workloads, and **Distributed** for enterprise-scale systems. You’ll also want to tune index parameters (e.g., HNSW) to balance speed and accuracy, and set up monitoring for latency and recall to ensure the system remains reliable under load.


### Multi-Agent Collaboration

Another powerful use case is multi-agent collaboration. Instead of one agent trying to do everything, you define several specialized agents that work together. In a software development workflow, for example, a Product Manager Agent breaks down requirements, an Architect Agent drafts the design, a Developer Agent writes code, and a Testing Agent validates the results.

This orchestration highlights LangGraph’s strengths—state management, branching, and coordination across agents. We’ll cover this setup in more detail in a later article, but the key point is that `langgraph-up-react` makes it practical to try these patterns without spending weeks on scaffolding.


## Conclusion

Building reliable agents isn’t just about clever prompts—it’s about structuring reasoning, managing state, and wiring everything into a system you can actually maintain. LangGraph gives you the framework to do that, and `langgraph-up-react` lowers the barrier by handling the boilerplate so you can focus on agent behavior.

With this template, you can spin up projects like knowledge base Q&A systems or multi-agent workflows without getting lost in setup. It’s a starting point that saves time, avoids common pitfalls, and makes experimenting with LangGraph far smoother.

In the next post, I’ll go deeper into a hands-on tutorial—showing step by step how to extend the template and build a working agent for a real use case using LangGraph, `langgraph-up-react`, and Milvus vector database. Stay tuned.


Getting Started with langgraph-up-react: A LangGraph Template


Getting Started with langgraph-up-react: A Practical LangGraph Template



When building with large language models (LLMs), the framework you choose has a huge impact on your development experience. A good framework streamlines workflows, reduces boilerplate, and makes it easier to move from prototype to production. A poor fit can do the opposite, adding friction and technical debt.

Two of the most popular options today are [**LangChain**](https://python.langchain.com/docs/introduction/) and [**LangGraph**](https://langchain-ai.github.io/langgraph/) — both open source and created by the LangChain team. LangChain focuses on component orchestration and workflow automation, making it a good fit for common use cases like retrieval-augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)). LangGraph builds on top of LangChain with a graph-based architecture, which is better suited for stateful applications, complex decision-making, and multi-agent coordination.

In this guide, we’ll compare the two frameworks side by side: how they work, their strengths, and the types of projects they’re best suited for. By the end, you’ll have a clearer sense of which one makes the most sense for your needs.


## LangChain: Your Component Library and LCEL Orchestration Powerhouse

[**LangChain**](https://github.com/langchain-ai/langchain) is an open-source framework designed to make building LLM applications more manageable. You can think of it as the middleware that sits between your model (say, OpenAI’s [GPT-5](https://milvus.io/blog/gpt-5-review-accuracy-up-prices-down-code-strong-but-bad-for-creativity.md) or Anthropic’s [Claude](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md)) and your actual app. Its main job is to help you _chain together_ all the moving parts: prompts, external APIs, [vector databases](https://zilliz.com/learn/what-is-vector-database), and custom business logic.

Take RAG as an example. Instead of wiring everything from scratch, LangChain gives you ready-made abstractions to connect an LLM with a vector store (like [Milvus](https://milvus.io/) or [Zilliz Cloud](https://zilliz.com/cloud)), run semantic search, and feed results back into your prompt. Beyond that, it offers utilities for prompt templates, agents that can call tools, and orchestration layers that keep complex workflows maintainable.

**What makes LangChain stand out?**

- **Rich component library** – Document loaders, text splitters, vector storage connectors, model interfaces, and more.

- **LCEL (LangChain Expression Language) orchestration** – A declarative way to mix and match components with less boilerplate.

- **Easy integration** – Works smoothly with APIs, databases, and third-party tools.

- **Mature ecosystem** – Strong documentation, examples, and an active community.


## LangGraph: Your Go-To for Stateful Agent Systems

[LangGraph](https://github.com/langchain-ai/langgraph) is a specialized extension of LangChain that focuses on stateful applications. Instead of writing workflows as a linear script, you define them as a graph of nodes and edges — essentially a state machine. Each node represents an action (like calling an LLM, querying a database, or checking a condition), while the edges define how the flow moves depending on the results. This structure makes it easier to handle loops, branching, and retries without your code turning into a tangle of if/else statements.

This approach is especially useful for advanced use cases such as copilots and [autonomous agents](https://zilliz.com/blog/what-exactly-are-ai-agents-why-openai-and-langchain-are-fighting-over-their-definition). These systems often need to keep track of memory, handle unexpected results, or make decisions dynamically. By modeling the logic explicitly as a graph, LangGraph makes these behaviors more transparent and maintainable.

**Core features of LangGraph include:**

- **Graph-based architecture** – Native support for loops, backtracking, and complex control flows.

- **State management** – Centralized state ensures context is preserved across steps.

- **Multi-agent support** – Built for scenarios where multiple agents collaborate or coordinate.

- **Debugging tools** – Visualization and debugging via LangSmith Studio to trace graph execution.


## LangChain vs LangGraph: Technical Deep Dive

### Architecture 

LangChain uses **LCEL (LangChain Expression Language)** to wire components together in a linear pipeline. It’s declarative, readable, and great for straightforward workflows like RAG.


```
# LangChain LCEL orchestration example
from langchain.prompts import ChatPromptTemplate
from langchain.chat_models import ChatOpenAI

prompt = ChatPromptTemplate.from_template("Please answer the following question: {question}")
model = ChatOpenAI()

# LCEL chain orchestration
chain = prompt | model

# Run the chain
result = chain.invoke({"question": "What is artificial intelligence?"})
```


LangGraph takes a different approach: workflows are expressed as a **graph of nodes and edges**. Each node defines an action, and the graph engine manages state, branching, and retries.

```
# LangGraph graph structure definition
from langgraph.graph import StateGraph
from typing import TypedDict

class State(TypedDict):
    messages: list
    current_step: str

def node_a(state: State) -> State:
    return {"messages": state["messages"] + ["Processing A"], "current_step": "A"}

def node_b(state: State) -> State:
    return {"messages": state["messages"] + ["Processing B"], "current_step": "B"}

graph = StateGraph(State)
graph.add_node("node_a", node_a)
graph.add_node("node_b", node_b)
graph.add_edge("node_a", "node_b")
```


Where LCEL gives you a clean linear pipeline, LangGraph natively supports loops, branching, and conditional flows. This makes it a stronger fit for **agent-like systems** or multi-step interactions that don’t follow a straight line.


### State Management

- **LangChain**: Uses Memory components for passing context. Works fine for simple multi-turn conversations or linear workflows.

- **LangGraph**: Uses a centralized state system that supports rollbacks, backtracking, and detailed history. Essential for long-running, stateful apps where context continuity matters.


### Execution Models

| **Feature**           | **LangChain**                  | **LangGraph**              |
| --------------------- | ------------------------------ | -------------------------- |
| Execution Mode        | Linear orchestration           | Stateful (Graph) Execution |
| Loop Support          | Limited Support                | Native Support             |
| Conditional Branching | Implemented via RunnableMap    | Native Support             |
| Exception Handling    | Implemented via RunnableBranch | Built-in Support           |
| Error Processing      | Chain-style Transmission       | Node-level Processing      |


## Real-World Use Cases: When to Use Each

Frameworks aren’t just about architecture — they shine in different situations. So the real question is: when should you reach for LangChain, and when does LangGraph make more sense? Let’s look at some practical scenarios.


### When LangChain Is Your Best Choice

#### 1. Straightforward Task Processing

LangChain is a great fit when you need to transform input into output without heavy state tracking or branching logic. For example, a browser extension that translates selected text:

```
# Implementing simple text translation using LCEL
from langchain.prompts import ChatPromptTemplate
from langchain.chat_models import ChatOpenAI

prompt = ChatPromptTemplate.from_template("Translate the following text to English: {text}")
model = ChatOpenAI()
chain = prompt | model

result = chain.invoke({"text": "Hello, World!"})
```


In this case, there’s no need for memory, retries, or multi-step reasoning — just efficient input-to-output transformation. LangChain keeps the code clean and focused.


#### 2. Foundation Components

LangChain provides rich basic components that can serve as building blocks for constructing more complex systems. Even when teams build with LangGraph, they often rely on LangChain’s mature components. The framework offers:

- **Vector store connectors** – Unified interfaces for databases like Milvus and Zilliz Cloud.

- **Document loaders & splitters** – For PDFs, web pages, and other content.

- **Model interfaces** – Standardized wrappers for popular LLMs.

This makes LangChain not only a workflow tool but also a reliable component library for larger systems.


### When LangGraph Is the Clear Winner

#### 1. Sophisticated Agent Development

LangGraph excels when you’re building advanced agent systems that need to loop, branch, and adapt. Here’s a simplified agent pattern:

```
# Simplified Agent system example
def agent(state):
    messages = state["messages"]
    # Agent thinks and decides next action
    action = decide_action(messages)
    return {"action": action, "messages": messages}

def tool_executor(state):
    # Execute tool calls
    action = state["action"]
    result = execute_tool(action)
    return {"result": result, "messages": state["messages"] + [result]}

# Build Agent graph
graph = StateGraph()
graph.add_node("agent", agent)
graph.add_node("tool_executor", tool_executor)
graph.add_edge("agent", "tool_executor")
graph.add_edge("tool_executor", "agent")
```


**Example:** GitHub Copilot X's advanced features perfectly demonstrate LangGraph's agent architecture in action. The system understands developer intent, breaks complex programming tasks into manageable steps, executes multiple operations in sequence, learns from intermediate results, and adapts its approach based on what it discovers along the way.


#### 2. Advanced Multi-Turn Conversation Systems

LangGraph's state management capabilities make it very suitable for building complex multi-turn conversation systems:

- **Customer service systems**: Capable of tracking conversation history, understanding context, and providing coherent responses

- **Educational tutoring systems**: Adjusting teaching strategies based on students' answer history

- **Interview simulation systems**: Adjusting interview questions based on candidates' responses

**Example:** Duolingo's AI tutoring system showcases this perfectly. The system continuously analyzes each learner's response patterns, identifies specific knowledge gaps, tracks learning progress across multiple sessions, and delivers personalized language learning experiences that adapt to individual learning styles, pace preferences, and areas of difficulty.


#### 3. Multi-Agent Collaboration Ecosystems

LangGraph natively supports ecosystems where multiple agents coordinate. Examples include:

- **Team collaboration simulation**: Multiple roles collaboratively completing complex tasks

- **Debate systems**: Multiple roles holding different viewpoints engaging in debate

- **Creative collaboration platforms**: Intelligent agents from different professional domains creating together

This approach has shown promise in research domains like drug discovery, where agents model different areas of expertise and combine results into new insights.


### Making the Right Choice: A Decision Framework

| **Project Characteristics**     | **Recommended Framework** | **Why**                                    |
| ------------------------------- | ------------------------- | ------------------------------------------ |
| Simple One-Time Tasks           | LangChain                 | LCEL orchestration is simple and intuitive |
| Text Translation/Optimization   | LangChain                 | No need for complex state management       |
| Agent Systems                   | LangGraph                 | Powerful state management and control flow |
| Multi-Turn Conversation Systems | LangGraph                 | State tracking and context management      |
| Multi-Agent Collaboration       | LangGraph                 | Native support for multi-node interaction  |
| Systems Requiring Tool Usage    | LangGraph                 | Flexible tool invocation flow control      |


## Conclusion

In most cases, LangChain and LangGraph are complementary, not competitors. LangChain gives you a solid foundation of components and LCEL orchestration — great for quick prototypes, stateless tasks, or projects that just need clean input-to-output flows. LangGraph steps in when your application outgrows that linear model and requires state, branching, or multiple agents working together.

- **Choose LangChain** if your focus is on straightforward tasks like text translation, document processing, or data transformation, where each request stands on its own.

- **Choose LangGraph** if you’re building multi-turn conversations, agent systems, or collaborative agent ecosystems where context and decision-making matter.

- **Mix both** for the best results. Many production systems start with LangChain’s components (document loaders, vector store connectors, model interfaces) and then add LangGraph to manage stateful, graph-driven logic on top.

Ultimately, it’s less about chasing trends and more about aligning the framework with your project’s genuine needs. Both ecosystems are evolving rapidly, driven by active communities and robust documentation. By understanding where each fits, you’ll be better equipped to design applications that scale — whether you’re building your first RAG pipeline with Milvus or orchestrating a complex multi-agent system.


LangChain vs LangGraph: A Developer's Guide to Choosing Your AI Frameworks



Nano Banana is going viral on social media right now and for a good reason! You’ve probably seen the images it generates or even tried it yourself. It’s a latest image generation model that turns plain text into collectible-grade figurine shots with startling precision and speed. 

Type in something like _“swap Elon’s hat and skirt,”_ and in about 16 seconds, you’ll get a photoreal result: shirt tucked, colors blended, accessories in place—no manual edits. No lag.

![](https://assets.zilliz.com/beach_side_668179b830.png)

I couldn’t resist testing it either. My prompt was:

_“Use the Nano Banana model to create a 1/7 scale commercialized figure of the character in the illustration, in a realistic style and environment. Place the figure on a computer desk, using a circular transparent acrylic base without any text. On the computer screen, display the ZBrush modeling process of the figure. Next to the screen, place a Bandai-style toy packaging box printed with the original artwork.”_

![](https://assets.zilliz.com/me_with_a_dress_506a0ebf39.png)

The result blew me away—it looked like a production prototype straight from a convention booth.

It’s no surprise that teams are already finding serious use cases for it. One of our customers, a mobile entertainment platform featuring gacha and dress-up gameplay, is developing a feature that allows players to upload photos and instantly dress up their avatars with in-game accessories. E-commerce brands are experimenting with “shoot once, reuse forever”: capturing a base model image and then generating endless outfit and hairstyle variations with AI, instead of re-shooting 20 times in the studio.

But here’s the catch—image generation alone doesn’t solve the whole problem. These systems also need **smart retrieval**: the ability to instantly find the right outfits, props, and visual elements from massive, unstructured media libraries. Without that, the generative model is guessing in the dark. What companies really need is a **multimodal RAG (retrieval-augmented generation) system**—where Nano Banana handles the creativity, and a powerful vector database handles the context.

That’s where **Milvus** comes in. As an open-source vector database, Milvus can index and search across billions of embeddings—images, text, audio, and beyond. Paired with Nano Banana, it becomes the backbone of a production-ready multimodal RAG pipeline: search, match, and generate at enterprise scale.

In the rest of this blog, we’ll walk through how to combine Nano Banana and Milvus to build an enterprise-ready multimodal RAG system—and why this pairing unlocks the next wave of AI applications.

## Building a Text-to-Image Retrieval Engine

For fast-moving consumer goods brands, game studios, and media companies, the bottleneck in AI image generation isn’t the model—it’s the mess.

Their archives are a swamp of unstructured data, including product shots, character assets, promotional videos, and outfit renders. And when you need to find “the red cape from last season’s Lunar drop,” good luck—traditional keyword-based search can’t handle that.

The solution? Build a **text-to-image retrieval system**.

Here’s the play: use [CLIP](https://openai.com/research/clip?utm_source=chatgpt.com) to embed both text and image data into vectors. Store those vectors in **Milvus**, the open-source vector database purpose-built for similarity search. Then, when a user types a description (“red silk cape with gold trim”), you hit the DB and return the top 3 most semantically similar images.

It’s fast. It’s scalable. And it turns your messy media library into a structured, queryable asset bank.

Here’s how to build it:

Install Dependencies

```
# Install necessary packages
%pip install --upgrade pymilvus pillow matplotlib
%pip install git+https://github.com/openai/CLIP.git
```
Import Necessary Libraries
```
import os
import clip
import torch
from PIL import Image
import matplotlib.pyplot as plt
from pymilvus import MilvusClient
from glob import glob
import math

print("All libraries imported successfully!")
```
Initialize Milvus Client
```
# Initialize Milvus client
milvus_client = MilvusClient(uri="http://localhost:19530",token="root:Miluvs")
print("Milvus client initialized successfully!")
```
Load CLIP Model
```
# Load CLIP model
model_name = "ViT-B/32"
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load(model_name, device=device)
model.eval()

print(f"CLIP model '{model_name}' loaded successfully, running on device: {device}")
print(f"Model input resolution: {model.visual.input_resolution}")
print(f"Context length: {model.context_length}")
print(f"Vocabulary size: {model.vocab_size}")
```
Output result:
```
CLIP model `ViT-B/32` loaded successfully, running on: cpu
 Model input resolution: 224
 Context length: 77
 Vocabulary size: 49,408
```
Define Feature Extraction Functions
```
def encode_image(image_path):
    """Encode image into normalized feature vector"""
    try:
        image = preprocess(Image.open(image_path)).unsqueeze(0).to(device)
        
        with torch.no_grad():
            image_features = model.encode_image(image)
            image_features /= image_features.norm(dim=-1, keepdim=True)  # Normalize
        
        return image_features.squeeze().cpu().tolist()
    except Exception as e:
        print(f"Error processing image {image_path}: {e}")
        return None
def encode_text(text):
    """Encode text into normalized feature vector"""
    text_tokens = clip.tokenize([text]).to(device)
    
    with torch.no_grad():
        text_features = model.encode_text(text_tokens)
        text_features /= text_features.norm(dim=-1, keepdim=True)  # Normalize
    
    return text_features.squeeze().cpu().tolist()

print("Feature extraction functions defined successfully!")
```
Create Milvus Collection
```
collection_name = "production_image_collection"
# If collection already exists, delete it
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)
    print(f"Existing collection deleted: {collection_name}")

# Create new collection
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=512,  # CLIP ViT-B/32 embedding dimension
    auto_id=True,  # Auto-generate ID
    enable_dynamic_field=True,  # Enable dynamic fields
    metric_type="COSINE"  # Use cosine similarity
)

print(f"Collection '{collection_name}' created successfully!")
print(f"Collection info: {milvus_client.describe_collection(collection_name)}")
```
Collection creation success output:

```
Existing collection deleted: production_image_collection
Collection 'production_image_collection' created successfully!
Collection info: {'collection_name': 'production_image_collection', 'auto_id': True, 'num_shards': 1, 'description': '', 'fields': [{'field_id': 100, 'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'params': {}, 'auto_id': True, 'is_primary': True}, {'field_id': 101, 'name': 'vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 512}}, {'field_id': 102, 'name': 'function': [], 'aliases': [], 'collection_id': 460508990706033544, 'consistency_level': 2, 'properties': {}, 'num_partitions': 1, 'enable_dynamic_field': True, 'created_timestamp': 460511723827494913, 'updated_timestamp': 460511723827494913}
```
Process and Insert Images
```
# Set image directory path
image_dir = "./production_image"
raw_data = []

# Get all supported image formats
image_extensions = ['*.jpg', '*.jpeg', '*.png', '*.JPEG', '*.JPG', '*.PNG']
image_paths = []

for ext in image_extensions:
    image_paths.extend(glob(os.path.join(image_dir, ext)))

print(f"Found {len(image_paths)} images in {image_dir}")

# Process images and generate embeddings
successful_count = 0
for i, image_path in enumerate(image_paths):
    print(f"Processing progress: {i+1}/{len(image_paths)} - {os.path.basename(image_path)}")
    
    image_embedding = encode_image(image_path)
    if image_embedding is not None:
        image_dict = {
            "vector": image_embedding,
            "filepath": image_path,
            "filename": os.path.basename(image_path)
        }
        raw_data.append(image_dict)
        successful_count += 1

print(f"Successfully processed {successful_count} images")
```
Image processing progress output:
```
Found 50 images in ./production_image
Processing progress: 1/50 - download (5).jpeg
Processing progress: 2/50 - images (2).jpeg
Processing progress: 3/50 - download (23).jpeg
Processing progress: 4/50 - download.jpeg
Processing progress: 5/50 - images (14).jpeg
Processing progress: 6/50 - images (16).jpeg
…
Processing progress: 44/50 - download (10).jpeg
Processing progress: 45/50 - images (18).jpeg
Processing progress: 46/50 - download (9).jpeg
Processing progress: 47/50 - download (12).jpeg
Processing progress: 48/50 - images (1).jpeg
Processing progress: 49/50 - download.png
Processing progress: 50/50 - images.png
Successfully processed 50 images
```
Insert Data into Milvus
```
# Insert data into Milvus
if raw_data:
    print("Inserting data into Milvus...")
    insert_result = milvus_client.insert(collection_name=collection_name, data=raw_data)
    
    print(f"Successfully inserted {insert_result['insert_count']} images into Milvus")
    print(f"Sample inserted IDs: {insert_result['ids'][:5]}...")  # Show first 5 IDs
else:
    print("No successfully processed image data to insert")
```

Define Search and Visualization Functions
```
def search_images_by_text(query_text, top_k=3):
    """Search images based on text query"""
    print(f"Search query: '{query_text}'")
    
    # Encode query text
    query_embedding = encode_text(query_text)
    
    # Search in Milvus
    search_results = milvus_client.search(
        collection_name=collection_name,
        data=[query_embedding],
        limit=top_k,
        output_fields=["filepath", "filename"]
    )
    
    return search_results[0]


def visualize_search_results(query_text, results):
    """Visualize search results"""
    num_images = len(results)
    
    if num_images == 0:
        print("No matching images found")
        return
    
    # Create subplots
    fig, axes = plt.subplots(1, num_images, figsize=(5*num_images, 5))
    fig.suptitle(f'Search Results: "{query_text}" (Top {num_images})', fontsize=16, fontweight='bold')
    
    # Handle single image case
    if num_images == 1:
        axes = [axes]
    
    # Display images
    for i, result in enumerate(results):
        try:
            img_path = result['entity']['filepath']
            filename = result['entity']['filename']
            score = result['distance']
            
            # Load and display image
            img = Image.open(img_path)
            axes[i].imshow(img)
            axes[i].set_title(f"{filename}\nSimilarity: {score:.3f}", fontsize=10)
            axes[i].axis('off')
            
            print(f"{i+1}. File: {filename}, Similarity score: {score:.4f}")
            
        except Exception as e:
            axes[i].text(0.5, 0.5, f'Error loading image\n{str(e)}',
                        ha='center', va='center', transform=axes[i].transAxes)
            axes[i].axis('off')
    
    plt.tight_layout()
    plt.show()

print("Search and visualization functions defined successfully!")
```

Execute Text-to-Image Search

```
# Example search 1
query1 = "a golden watch"
results1 = search_images_by_text(query1, top_k=3)
visualize_search_results(query1, results1)
```




Search query execution output:

```
Search query: 'a golden watch'
1. File: images (19).jpeg, Similarity score: 0.2934
2. File: download (26).jpeg, Similarity score: 0.3073
3. File: images (17).jpeg, Similarity score: 0.2717
```


![](https://assets.zilliz.com/watch_067c39ba51.png)

## Using Nano-banana to Create Brand Promotional Images


Now that we have our text-to-image search system working with Milvus, let's integrate Nano-banana to generate new promotional content based on the assets we retrieve.

Install Google SDK
```
%pip install google-generativeai
%pip install requests
print("Google Generative AI SDK installation complete!")
```

Configure Gemini API

```
import google.generativeai as genai
from PIL import Image
from io import BytesIO
genai.configure(api_key="<your_api_key>")
```
Generate New Images


```
prompt = (
    "An European male model wearing a suit, carrying a gold watch."
)

image = Image.open("/path/to/image/watch.jpg")

model = genai.GenerativeModel('gemini-2.5-flash-image-preview')
response = model.generate_content([prompt, image])

for part in response.candidates[0].content.parts:
    if part.text is not None:
        print(part.text)
    elif part.inline_data is not None:
        image = Image.open(BytesIO(part.inline_data.data))
        image.save("generated_image.png")
        image.show()
```

![](https://assets.zilliz.com/suit_976b6f1df2.png)

## What This Means for Your Development Workflow

As a developer, this Milvus + Nano-banana integration fundamentally changes how you can approach content generation projects. Instead of managing static asset libraries or relying on expensive creative teams, you now have a dynamic system that retrieves and generates exactly what your application needs in real-time.

Consider the following recent client scenario: a brand launched several new products but opted to skip the traditional photo shoot pipeline entirely. Using our integrated system, they could instantly generate promotional imagery by combining their existing product database with Nano-banana's generation capabilities.


*Prompt: A model is wearing these products on the beach*

![](https://assets.zilliz.com/model_5a2a042b46.png)

The real power becomes apparent when you need to create complex, multi-variant content that would traditionally require extensive coordination between photographers, models, and set designers. With Milvus handling the asset retrieval and Nano-banana managing the generation, you can programmatically create sophisticated scenes that adapt to your specific requirements:



*Prompt: A model is posing and leaning against a blue convertible sports car. She is wearing a halter top dress and the accompanying accessories. She is adorned with a diamond necklace and a blue watch, wearing high heels on her feet and holding a labubu pendant in her hand.*

![](https://assets.zilliz.com/shoes_98e1e4c70b.png)

For developers working in gaming or collectibles, this system opens up entirely new possibilities for rapid prototyping and concept validation. Instead of investing weeks in 3D modeling before knowing if a concept works, you can now generate photorealistic product visualizations that include packaging, environmental context, and even manufacturing processes:

*Prompt: Use the nano-banana model to create a 1/7 scale commercialized figure of the character in the illustration, in a realistic style and environment. Place the figure on a computer desk, using a circular transparent acrylic base without any text. On the computer screen, display the ZBrush modeling process of the figure. Next to the computer screen, place a BANDAI-style toy packaging box printed with the original artwork.*

![](https://assets.zilliz.com/milvus_3d_5189d53773.png)

## Conclusion

From a technical perspective, Nano Banana is more than a novelty—it’s production-ready in ways that matter to developers. Its biggest strength is consistency and controllability, which means fewer edge cases bleeding into your application logic. Just as importantly, it handles the subtle details that often derail automated pipelines: keeping brand colors consistent, generating physically plausible lighting and reflections, and ensuring visual coherence across multiple output formats.

The real magic happens when you combine it with the Milvus vector database. A vector database doesn’t just store embeddings—it becomes an intelligent asset manager that can surface the most relevant historical content to guide new generations. The result: faster generation times (because the model has better context), higher consistency across your application, and the ability to enforce brand or style guidelines automatically. 

In short, Milvus transforms Nano Banana from a creative toy into a scalable enterprise system.

Of course, no system is flawless. Complex, multi-step instructions can still cause hiccups, and lighting physics sometimes stretches reality more than you’d like. The most reliable solution we’ve seen is to supplement text prompts with reference images stored in Milvus, which gives the model richer grounding, more predictable results, and shorter iteration cycles. With this setup, you’re not just experimenting with multimodal RAG—you’re running it in production with confidence.


Nano Banana + Milvus: Turning Hype into Enterprise-Ready Multimodal RAG


Hybrid Document Retrieval System with LangExtract + Milvus


LangExtract + Milvus: A Practical Guide to Building a Hybrid Document Processing and Search System


Why I’m Against Claude Code’s Grep-Only Retrieval? It Just Burns Too Many Tokens



Remember when working with data meant crafting SQL queries for exact matches? Those days are long gone. We’ve entered the era of AI and semantic search, where AI doesn’t just match keywords—it understands intent. And at the heart of this shift are vector databases: the engines powering today’s most advanced applications, from ChatGPT’s retrieval systems to Netflix’s personalized recommendations to Tesla’s autonomous driving stack.

But here's the plot twist: not all [vector databases ](https://zilliz.com/learn/what-is-vector-database)are created equal. 

Your RAG application needs lightning-fast semantic retrieval across billions of documents. Your recommendation system demands sub-millisecond responses under crushing traffic loads. Your computer vision pipeline requires handling exponentially growing image datasets without breaking the bank.

Meanwhile, the market is flooded with options: Elasticsearch, Milvus, PGVector, Qdrant, and even AWS’s new S3 Vector. Each claims to be the best—but the best for what? Choosing wrong could mean wasted months of engineering, runaway infrastructure costs, and a serious hit to your product’s competitive edge.

That’s where this guide comes in. Instead of vendor hype, we’ll walk through a practical decision framework across three critical dimensions: functionality, performance, and ecosystem. By the end, you’ll have the clarity to choose the database that’s not just “popular,” but the one that’s right for your use case.


## 1. Functionality: Can It Handle Your AI Workload?

When choosing a vector database, functionality is the foundation. It’s not just about storing vectors—it’s about whether the system can support the diverse, large-scale, and often messy requirements of real-world AI workloads. You’ll need to evaluate both core vector capabilities and enterprise-grade features that determine long-term viability.


### Complete Vector Data Type Support

Different AI tasks generate different kinds of vectors—text, images, audio, and user behavior. A production system often needs to handle them all at once. Without full support for multiple vector types, your database won’t even make it past day one.

Take an e-commerce product search as an example:

- Product images → dense vectors for visual similarity and image-to-image search.

- Product descriptions → sparse vectors for keyword matching and full-text retrieval.

- User behavior patterns (clicks, purchases, favorites) → binary vectors for fast matching of interests.

On the surface, it looks like “search,” but under the hood, it’s a multi-vector, multimodal retrieval problem.

![](https://assets.zilliz.com/20250822_192755_c6c0842b05.png)


### Rich Indexing Algorithms with Fine-Grained Control

Every workload forces a trade-off between recall, speed, and cost—the classic “impossible triangle.” A robust vector database should offer multiple indexing algorithms so you can choose the right compromise for your use case:

- Flat → highest accuracy, at the cost of speed.

- IVF → scalable, high-performance retrieval for large datasets.

- HNSW → strong balance between recall and latency.

Enterprise-grade systems also go further with:

- Disk-based indexing for petabyte-scale storage at lower cost.

- GPU acceleration for ultra-low-latency inference.

- Granular parameter tuning so teams can optimize every query path to business requirements.

The best systems also provide granular parameter tuning, letting you squeeze optimal performance from limited resources and fine-tune indexing behavior to match your specific business requirements.


### Comprehensive Retrieval Methods

Top-K similarity search is table stakes. Real applications demand more sophisticated retrieval strategies, such as filtering retrieval (price ranges, stock status, thresholds), grouping retrieval (category diversity, e.g., dresses vs. skirts vs. suits), and hybrid retrieval (combining sparse text with dense image embeddings as well as full-text search).

For example, a simple “show me dresses” request on an e-commerce site may trigger:

1. Similarity retrieval on product vectors (image + text).

2. Scalar filtering for price and stock availability.

3. Diversity optimization to surface varied categories.

4. Hybrid personalization blending user profile embeddings with purchase history.

What looks like a simple recommendation is actually powered by a retrieval engine with layered, complementary capabilities.

![](https://assets.zilliz.com/recsyc_da5d86d6f4.png)


### Enterprise-Grade Architecture

Unstructured data is exploding. According to IDC, by 2027 it will reach 246.9 zettabytes—an astonishing 86.8% of all global data. Once you start processing that volume through AI models, you’re dealing with astronomical amounts of vector data that only grow faster over time.

A vector database built for hobby projects won’t survive this curve. To succeed at enterprise scale, you need a database with cloud-native flexibility and scalability baked in. That means:

- Elastic scaling to handle unpredictable spikes in workload.

- Multi-tenant support so teams and applications can share infrastructure securely.

- Seamless integration with Kubernetes and cloud services for automated deployment and scaling.

And because downtime is never acceptable in production, resilience is just as critical as scalability. Enterprise-ready systems should provide:

- High availability with automatic failover.

- Multi-replica disaster recovery across regions or zones.

- Self-healing infrastructure that detects and corrects failures without human intervention.

In short: handling vectors at scale isn’t just about fast queries—it’s about an architecture that grows with your data, protects against failure, and stays cost-efficient at enterprise volumes.


## 2. Performance: Will It Scale When Your App Goes Viral?

Once functionality is covered, performance becomes the make-or-break factor. The right database must not only handle today’s workloads but also scale gracefully when traffic spikes. Evaluating performance means looking at multiple dimensions—not just raw speed.


### Key Performance Metrics

The Complete Vector Database Evaluation Framework covers:

- Latency (P50, P95, P99) → captures both average and worst-case response times.

- Throughput (QPS) → measures concurrency under real-world loads.

- Accuracy (Recall@K) → ensures approximate search still returns relevant results.

- Data scale adaptability → tests performance at millions, tens of millions, and billions of records.

Beyond Basic Metrics: In production, you’ll also want to measure:

- Filtered query performance across varying ratios (1%–99%).

- Streaming workloads with continuous inserts + real-time queries.

- Resource efficiency (CPU, memory, disk I/O) to ensure cost-effectiveness.


### Benchmarking in Practice

While[ ANN-Benchmark](http://ann-benchmarks.com/) offers widely-recognized algorithm-level evaluation, it focuses on underlying algorithm libraries and misses dynamic scenarios. The datasets feel outdated, and the use cases are too simplified for production environments.

For real-world vector database evaluation, we recommend the open-source[ VDBBench](https://github.com/zilliztech/VectorDBBench), which tackles the complexities of production testing with comprehensive scenario coverage.

A solid VDBBench testing approach follows three essential steps:

- Determine use scenarios by selecting appropriate datasets (like SIFT1M or GIST1M) and business scenarios (TopK retrieval, filtered retrieval, concurrent write-and-read operations)

- Configure database and VDBBench parameters to ensure fair, reproducible testing environments

- Execute and analyze tests through the web interface to automatically collect performance metrics, compare results, and make data-driven selection decisions

For more information about how to benchmark a vector database with real-world workloads, check this tutorial: [How to Evaluate VectorDBs that Match Production via VDBBench ](https://milvus.io/blog/hands-on-with-vdbbench-benchmarking-vector-databases-for-pocs-that-match-production.md)


## 3. Ecosystem: Is It Ready for Production Reality?

A vector database doesn’t live in isolation. Its ecosystem determines how easy it is to adopt, how quickly it scales, and whether it can survive in production over the long run. When evaluating, it helps to look at four key dimensions. 


(1) Fit with the AI Ecosystem

A top-tier and production-ready vector database should plug directly into the AI tools you already use. That means:

- Native support for mainstream LLMs (OpenAI, Claude, Qwen) and embedding services.

- Compatibility with development frameworks like LangChain, LlamaIndex, and Dify, so you can build RAG pipelines, recommendation engines, or Q&A systems without fighting the stack.

- Flexibility in handling vectors from multiple sources—text, images, or custom models.

(2) Tooling That Supports Daily Operations

The best vector database in the world won’t succeed if it’s painful to operate. Look for a vector database that is seamlessly compatible with the surrounding tool ecosystem that covers:

- Visual dashboards for managing data, monitoring performance, and handling permissions.

- Backup & recovery with both full and incremental options.

- Capacity planning tools that help forecast resources and scale clusters efficiently.

- Diagnostics & tuning for log analysis, bottleneck detection, and troubleshooting.

- Monitoring & alerts via standard integrations like Prometheus and Grafana.

These aren’t “nice to haves”—they’re what keep your system stable at 2 a.m. when traffic spikes.

(3) Open Source + Commercial Balance

Vector databases are still evolving. Open source brings speed and community feedback, but large-scale projects also need sustainable commercial backing. The most successful data platforms—think Spark, MongoDB, Kafka—all balance open innovation with strong companies behind them.

Commercial offerings should also be cloud-neutral: elastic, low-maintenance, and flexible enough to meet different business needs across industries and geographies.

(4) Proof in Real Deployments

Marketing slides don’t mean much without real customers. A credible vector database should have case studies across industries—finance, healthcare, manufacturing, internet, legal—and across use cases like search, recommendation, risk control, customer support, and quality inspection.

If your peers are already succeeding with it, that’s the best sign you can. And when in doubt, nothing beats running a proof of concept with your own data.


## Milvus: The Most Popular Open-Source Vector Database

If you’ve applied the evaluation framework—functionality, performance, ecosystem—you’ll find only a few vector databases that consistently deliver across all three dimensions. [Milvus](https://milvus.io/) is one of them.

Born as an open-source project and backed by [Zilliz](https://zilliz.com/), [Milvus](https://milvus.io/) is purpose-built for AI-native workloads. It combines advanced indexing and retrieval with enterprise-grade reliability, while still being approachable for developers building RAG, AI Agents, recommendation engines, or semantic search systems. With [36K+ GitHub](https://github.com/milvus-io/milvus) stars and adoption by more than 10,000 enterprise companies, Milvus has become the most popular open-source vector database in production today.

Milvus also provides multiple [deployment options](https://milvus.io/docs/install-overview.md), all under a single API:

- **Milvus Lite** → lightweight version for rapid experimentation and prototyping.

- **Standalone** → simple production deployments.

- **Cluster** → distributed deployments that scale to billions of vectors.

This deployment flexibility means teams can start small and scale seamlessly—without rewriting a single line of code.

Key capabilities at a glance: 

- 🔎**Comprehensive functionality** → Multimodal vector support (text, image, audio, etc.), multiple indexing methods (IVF, HNSW, disk-based, GPU acceleration), and advanced retrieval (hybrid, filtered, grouped, and full-text search).

- ⚡**Proven performance** → Tuned for billion-scale datasets, with adjustable indexing and benchmarking via tools like VDBBench.

- 🌐**Robust ecosystem** → Tight integrations with LLMs, embeddings, and frameworks like LangChain, LlamaIndex, and Dify. Includes a full operational toolchain for monitoring, backup, recovery, and capacity planning.

- 🛡️**Enterprise ready** → High availability, multi-replica disaster recovery, RBAC, observability, plus **Zilliz Cloud** for fully managed, cloud-neutral deployments.

Milvus gives you the flexibility of open source, the scale and reliability of enterprise systems, and the ecosystem integrations needed to move fast in AI development. It’s no surprise that it has become the go-to vector database for both startups and global enterprises.


### If You Want Zero Hassle—Try Zilliz Cloud (Managed Milvus)

Milvus is open source and always free to use. But if you’d rather focus on innovation instead of infrastructure, consider [Zilliz Cloud](https://zilliz.com/cloud)—the fully managed Milvus service built by the original Milvus team. It gives you everything you love about Milvus, plus advanced enterprise-grade features, without the operational overhead.

Why Teams Choose Zilliz Cloud? Key capabilities at a glance: 

- ⚡ **Deploy in minutes, scale automatically** 

- 💰 **Pay only for what you use** 

- 💬 **Natural language querying** 

- 🔒 **Enterprise-grade security**

- 🌍 **Global scale, local performance** 

- 📈 **99.95% uptime SLA**

For startups and enterprises alike, the value is clear: your technical teams should spend their time building products, not managing databases. Zilliz Cloud takes care of the scaling, security, and reliability—so you can pay 100% of your effort on delivering breakthrough AI applications.


## Choose Wisely: Your Vector Database Will Shape Your AI Future

Vector databases are evolving at breakneck speed, with new features and optimizations emerging almost monthly. The framework we’ve outlined—functionality, performance, and ecosystem—gives you a structured way to cut through the noise and make informed decisions today. But adaptability is just as important, since the landscape will keep shifting.

The winning approach is systematic evaluation backed by hands-on testing. Use the framework to narrow your choices, then validate with a proof-of-concept on your own data and workloads. That combination of rigor and real-world validation is what separates successful deployments from costly mistakes.

As AI applications grow more sophisticated and data volumes surge, the vector database you choose now will likely become a cornerstone of your infrastructure. Investing the time to evaluate thoroughly today will pay off in performance, scalability, and team productivity tomorrow.

In the end, the future belongs to teams that can harness semantic search effectively. Choose your vector database wisely—it may be the competitive advantage that sets your AI applications apart.


Guide | How to Choose the Right VectorDB for Your AI Apps


A Practical Guide for Choosing the Right Vector Database for Your AI Applications



## Introduction

Vibe Coding is transforming how we write software. Tools like Cursor and Windsurf are making development feel effortless and intuitive—ask for a function and get a snippet, need a quick API call, and it's generated before you finish typing. The promise is smooth, seamless development where your AI assistant anticipates your needs and delivers exactly what you want.

But there's a critical flaw breaking this beautiful flow: AI assistants frequently generate outdated code that breaks in production.

Consider this example: I asked Cursor to generate Milvus connection code, and it produced this:

```
connections.connect("default", host="localhost", port="19530")
```

This used to work perfectly, but the current pymilvus SDK recommends using `MilvusClient` for all connections and operations. The old method is no longer considered best practice, yet AI assistants continue to suggest it because their training data is often months or years out of date.

Despite all the progress in Vibe Coding tools, developers still spend significant time bridging the “last mile” between generated code and production-ready solutions. The vibe is there, but the accuracy isn’t.


### What is the Milvus SDK Code Helper?

The **Milvus SDK Code Helper** is a developer-focused solution that solves the _“last mile”_ problem in Vibe Coding—bridging the gap between AI-assisted coding and production-ready Milvus applications.

At its core, it’s a **Model Context Protocol (MCP) server** that connects your AI-powered IDE directly to the latest official Milvus documentation. Combined with Retrieval-Augmented Generation (RAG), it ensures the code your assistant generates is always accurate, up-to-date, and aligned with the Milvus best practices.

Instead of outdated snippets or guesswork, you get context-aware, standards-compliant code suggestions—right inside your development workflow.

**Key Benefits:**

- ⚡ **Configure once, boost efficiency forever**: Set it up once and enjoy consistently updated code generation

- 🎯 **Always current**: Access to the latest official Milvus SDK documentation

- 📈 **Improved code quality**: Generate code that follows current best practices

- 🌊 **Restored flow**: Keep your Vibe Coding experience smooth and uninterrupted

**Three Tools in One**

1. `pymilvus-code-generator` → Quickly write Python code for common Milvus tasks (e.g., creating collections, inserting data, running vector searches).

2. `orm-client-code-converter` → Modernize legacy Python code by replacing outdated ORM patterns with the latest `MilvusClient` syntax.

3. `language-translator` → Seamlessly convert Milvus SDK code between languages (e.g., Python ↔ TypeScript).

Check the resources below for more details: 

- Blog: [Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP ](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)

- Doc: [Milvus SDK Code Helper Guide | Milvus Documentation](https://milvus.io/docs/milvus-sdk-helper-mcp.md#Quickstart)


### Before You Begin

Before diving into the setup process, let's examine the dramatic difference the Code Helper makes in practice. The comparison below shows how the same request for creating a Milvus collection produces completely different results:

| **MCP Code Helper Enabled:**                                                                                                                                                                                              | **MCP Code Helper Disabled:**                                                                                                                                                                                             |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ![](https://assets.zilliz.com/mcp_enabled_fcb94737fb.png) | ![](https://assets.zilliz.com/mcp_disabled_769db4faee.png) |

This perfectly illustrates the core problem: without the Code Helper, even the most advanced AI assistants generate code using outdated ORM SDK patterns that are no longer recommended. The Code Helper ensures you get the most current, efficient, and officially endorsed implementation every time.

**The Difference in Practice:**

- **Modern approach**: Clean, maintainable code using current best practices

- **Deprecated approach**: Code that works but follows outdated patterns

- **Production impact**: Current code is more efficient, easier to maintain, and future-proof

This guide will walk you through setting up the Milvus SDK Code Helper across multiple AI IDEs and development environments. The setup process is straightforward and typically takes just a few minutes per IDE.


## Setting Up the Milvus SDK Code Helper

The following sections provide detailed setup instructions for each supported IDE and development environment. Choose the section that corresponds to your preferred development setup.


### Cursor IDE Setup

Cursor offers seamless integration with MCP servers through its built-in configuration system.

**Step 1: Access MCP Settings** 

Navigate to: Settings → Cursor Settings → Tools & Integrations → Add new global MCP server

![](https://assets.zilliz.com/cursor_mcp_configuration_interface_9ff0b7dcb7.png) _Cursor MCP configuration interface_

**Step 2: Configure the MCP Server** 

You have two options for configuration:

**Option A: Global Configuration (Recommended)** 

Add the following configuration to your Cursor `~/.cursor/mcp.json` file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Option B: Project-Specific Configuration** 

Create a `.cursor/mcp.json` file in your project folder with the same configuration above.

For additional configuration options and troubleshooting, refer to the[ Cursor MCP documentation](https://docs.cursor.com/context/model-context-protocol).


### Claude Desktop Setup

Claude Desktop provides straightforward MCP integration through its configuration system.

**Step 1: Locate Configuration File** 

Add the following configuration to your Claude Desktop configuration file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 2: Restart Claude Desktop** 

After saving the configuration, restart Claude Desktop to activate the new MCP server.


### Claude Code Setup

Claude Code offers command-line configuration for MCP servers, making it ideal for developers who prefer terminal-based setup.

**Step 1: Add MCP Server via Command Line** 

Execute the following command in your terminal:

```
claude mcp add-json sdk-code-helper --json '{
  "url": "https://sdk.milvus.io/mcp/",
  "headers": {
    "Accept": "text/event-stream"
  }
}
```

**Step 2: Verify Installation** 

The MCP server will be automatically configured and ready for use immediately after running the command.


### Windsurf IDE Setup

Windsurf supports MCP configuration through its JSON-based settings system.

**Step 1: Access MCP Settings** 

Add the following configuration to your Windsurf MCP settings file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 2: Apply Configuration** 

Save the settings file and restart Windsurf to activate the MCP server.


### VS Code Setup

VS Code integration requires an MCP-compatible extension to function properly.

**Step 1: Install MCP Extension** 

Ensure you have an MCP-compatible extension installed in VS Code.

**Step 2: Configure MCP Server** 

Add the following configuration to your VS Code MCP settings:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```


### Cherry Studio Setup

Cherry Studio provides a user-friendly graphical interface for MCP server configuration, making it accessible for developers who prefer visual setup processes.

**Step 1: Access MCP Server Settings** 

Navigate to Settings → MCP Servers → Add Server through the Cherry Studio interface.

**Step 2: Configure Server Details** 

Fill in the server configuration form with the following information:

- **Name**: `sdk code helper`

- **Type**: `Streamable HTTP`

- **URL**: `https://sdk.milvus.io/mcp/`

- **Headers**: `"Accept": "text/event-stream"`

**Step 3: Save and Activate** 

Click Save to activate the server configuration.

![](https://assets.zilliz.com/cherry_studio_mcp_configuration_interface_b7dce8b26d.png) 

_Cherry Studio MCP configuration interface_


### Cline Setup

Cline uses a JSON-based configuration system accessible through its interface.

**Step 1: Access MCP Settings**

1. Open Cline and click on the MCP Servers icon in the top navigation bar

2. Select the Installed tab

3. Click Advanced MCP Settings

**Step 2: Edit Configuration File** In the `cline_mcp_settings.json` file, add the following configuration:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 3: Save and Restart** 

Save the configuration file and restart Cline to apply the changes.


### Augment Setup

Augment provides access to MCP configuration through its advanced settings panel.

**Step 1: Access Settings**

1. Press Cmd/Ctrl + Shift + P or navigate to the hamburger menu in the Augment panel

2. Select Edit Settings

3. Under Advanced, click Edit in settings.json

**Step 2: Add Server Configuration** 

Add the server configuration to the `mcpServers` array in the `augment.advanced` object:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```


### Gemini CLI Setup

Gemini CLI requires manual configuration through a JSON settings file.

**Step 1: Create or Edit Settings File**

 Create or edit the `~/.gemini/settings.json` file on your system.

**Step 2: Add Configuration**

 Insert the following configuration into the settings file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 3: Apply Changes** 

Save the file and restart Gemini CLI to apply the configuration changes.


### Roo Code Setup

Roo Code utilizes a centralized JSON configuration file for managing MCP servers.

**Step 1: Access Global Configuration**

1. Open Roo Code

2. Navigate to Settings → MCP Servers → Edit Global Config

**Step 2: Edit Configuration File** 

In the `mcp_settings.json` file, add the following configuration:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 3: Activate Server** 

Save the file to automatically activate the MCP server.


### Verification and Testing

After completing the setup for your chosen IDE, you can verify that the Milvus SDK Code Helper is working correctly by:

1. **Testing Code Generation**: Ask your AI assistant to generate Milvus-related code and observe whether it uses current best practices

2. **Checking Documentation Access**: Request information about specific Milvus features to ensure the helper is providing up-to-date responses

3. **Comparing Results**: Generate the same code request with and without the helper to see the difference in quality and currentness


## Conclusion

By setting up the Milvus SDK Code Helper, you've taken a crucial step toward the future of development—where AI assistants generate not just fast code, but **accurate, current code**. Instead of relying on static training data that becomes obsolete, we're moving toward dynamic, real-time knowledge systems that evolve with the technologies they support.

As AI coding assistants become more sophisticated, the gap between tools with current knowledge and those without will only widen. The Milvus SDK Code Helper is just the beginning—expect to see similar specialized knowledge servers for other major technologies and frameworks. The future belongs to developers who can harness AI's speed while ensuring accuracy and currentness. You're now equipped with both.


Milvus SDK Code Helper Tutorial: Fix AI Outdated Code


Stop Your AI Assistant from Writing Outdated Code with Milvus SDK Code Helper


Tutorial | How to Evaluate VectorDBs that Match Production via VDBBench


Hands-On with VDBBench: Benchmarking Vector Databases for POCs That Match Production



AI agents are everywhere right now—from coding copilots to customer service bots—and they can be jaw-droppingly good at complex reasoning. Like many of you, I love them. But after building both agents and automation workflows, I’ve learned a simple truth: **agents aren’t the best solution for every problem.** 

For example, when I built a multi-agent system with CrewAI for decoding ML, things got messy fast. Research agents ignored web crawlers 70% of the time. Summary agents dropped citations. Coordination fell apart whenever tasks weren’t crystal clear.

And it’s not just in experiments. Many of us are already bouncing between ChatGPT for brainstorming, Claude for coding, and a half-dozen APIs for data processing—quietly thinking: _there has to be a better way to make all this work together._

Sometimes, the answer is an agent. More often, it’s a **well-designed AI workflow** that stitches your existing tools into something powerful, without the unpredictable complexity. 


## Building Smarter AI Workflows with Refly and Milvus

I know some of you are already shaking your heads: “Workflows? Those are rigid. They’re not smart enough for real AI automation.” Fair point—most workflows are rigid, because they’re modeled after old-school assembly lines: step A → step B → step C, no deviation allowed.

But the real issue isn’t the _idea_ of workflows—it’s the _execution_. We don’t have to settle for brittle, linear pipelines. We can design smarter workflows that adapt to context, flex with creativity, and still deliver predictable results. 

In this guide, we’ll build a complete content creation system using Refly and Milvus to show why AI workflows can outperform complex multi-agent architectures, especially if you care about speed, reliability, and maintainability.


### The Tools We’re Using

[**Refly**](https://refly.ai/): An open-source, AI-native content creation platform built around a “free canvas” concept.

- **Core capabilities:** intelligent canvas, knowledge management, multi-threaded dialogue, and professional creation tools.

- **Why it’s useful:** Drag-and-drop workflow building lets you chain tools together into cohesive automation sequences, without locking you into rigid, single-path execution.

[**Milvus**](https://milvus.io/): An open-source vector database handling the data layer.

- **Why it matters:** Content creation is mostly about finding and recombining existing information. Traditional databases handle structured data well, but most creative work involves unstructured formats—documents, images, videos.

- **What it adds:** Milvus leverages integrated embedding models to encode unstructured data as vectors, enabling semantic search so your workflows can retrieve relevant context with millisecond latency. Through protocols like MCP, it integrates seamlessly with your AI frameworks, letting you query data in natural language instead of wrestling with database syntax.


### Setting Up Your Environment

Let me walk you through setting this workflow up locally. 

**Quick setup checklist:**

- Ubuntu 20.04+ (or similar Linux)

- Docker + Docker Compose

- An API key from any LLM that supports function calling. Here in this guide, I’ll use [Moonshot](https://platform.moonshot.ai/docs/introduction#text-generation-model)’s LLM. 

**System Requirements**

- CPU: 8 cores minimum (16 cores recommended)

- Memory: 16GB minimum (32GB recommended)

- Storage: 100GB SSD minimum (500GB recommended)

- Network: Stable internet connection required

**Software Dependencies**

- Operating System: Linux (Ubuntu 20.04+ recommended)

- Containerization: Docker + Docker Compose

- Python: Version 3.11 or higher

- Language Model: Any model supporting function calls (online services or Ollama offline deployment both work)


### Step 1: Deploy the Milvus Vector Database

**1.1 Download Milvus**

```
wget https://github.com/milvus-io/milvus/releases/download/v2.5.12/milvus-standalone-docker-compose.yml -O docker-compose.yml
```

**1.2 Launch Milvus services**

```
docker-compose up -d
docker-compose ps -a
```

![](https://assets.zilliz.com/1_b93ce78614.png)


### Step 2: Deploy the Refly Platform

**2.1 Clone the repository**

You can use default values for all environment variables unless you have specific requirements:

```
git clone https://github.com/refly-ai/refly.git
```
```
cd deploy/docker
cp ../../apps/api/.env.example .env # copy the example api env file
docker compose up -d
```

**2.2 Verify service status**

```
docker ps -a
```

![](https://assets.zilliz.com/2_cfcde2c570.png)

### Step 3: Set Up MCP Services

**3.1 Download the Milvus MCP server**

```
git clone https://github.com/zilliztech/mcp-server-milvus.git
cd mcp-server-milvus
```

**3.2 Start the MCP service**

This example uses SSE mode. Replace the URI with your available Milvus service endpoint:

```
uv run src/mcp_server_milvus/server.py --sse --milvus-uri http://localhost:19530 --port 8000
```

**3.3 Confirm MCP service is running**

![](https://assets.zilliz.com/3_b755922c41.png)


### Step 4: Configuration and Setup

Now that your infrastructure is running, let's configure everything to work together seamlessly.

**4.1 Access the Refly platform**

Navigate to your local Refly instance:

```
http://192.168.7.148:5700
```

![](https://assets.zilliz.com/4_c1e421fece.png)

**4.2 Create your account**

![](https://assets.zilliz.com/42_4b8af22fe3.png)

![](https://assets.zilliz.com/422_e8cd8439f0.png)

**4.3 Configure your language model**

For this guide, we'll use [Moonshot](https://platform.moonshot.ai/docs/introduction#text-generation-model). First, register and obtain your API key.

**4.4 Add your model provider**

Enter the API key you obtained in the previous step:

![](https://assets.zilliz.com/44_b085f9a263.png)

**4.5 Configure the LLM model**

Make sure to select a model that supports function calling capabilities, as this is essential for the workflow integrations we'll be building:

![](https://assets.zilliz.com/45_a05213d0fa.pngQ)

**4.6 Integrate Milvus-MCP service**

Note that the web version doesn't support stdio-type connections, so we'll use the HTTP endpoint we set up earlier:

![](https://assets.zilliz.com/46_027e21e479.png)

![](https://assets.zilliz.com/462_959ee44a78.png)

Excellent! With everything configured, let's see this system in action through some practical examples.

**4.7 Example: Efficient Vector Retrieval with MCP-Milvus-Server**

This example shows how the **MCP-Milvus-Server** works as middleware between your AI models and Milvus vector database instances. It acts like a translator—accepting natural language requests from your AI model, converting them into the right database queries, and returning the results—so your models can work with vector data without knowing any database syntax.

**4.7.1 Create a new canvas**

![](https://assets.zilliz.com/471_a684e275ed.png)

**4.7.2 Start a conversation**

Open the dialogue interface, select your model, input your question, and send. 

**4.7.3 Review the results**

![](https://assets.zilliz.com/473_7c24a28999.png)

What’s happening here is pretty remarkable: we’ve just shown natural language control of a Milvus vector database using [MCP-Milvus-Server](https://milvus.io/blog/talk-to-vector-db-manage-milvus-via-natural-language.md) as the integration layer. No complex query syntax—just tell the system what you need in plain English, and it handles the database operations for you. 

**4.8 Example 2: Building a Refly Deployment Guide with Workflows**

This second example shows the real power of workflow orchestration. We'll create a complete deployment guide by combining multiple AI tools and data sources into a single, coherent process.

**4.8.1 Gather your source materials**

The power of Refly is its flexibility in handling different input formats. You can import resources in multiple formats, whether they're documents, images, or structured data.

**4.8.2 Create tasks and link resource cards**

Now we'll create our workflow by defining tasks and connecting them to our source materials.

**4.8.3 Set up three processing tasks**

This is where the workflow approach really shines. Instead of trying to handle everything in one complex process, we break the work into three focused tasks that integrate uploaded materials and refine them systematically.

- **Content integration task**: Combines and structures source materials

- **Content refinement task**: Improves clarity and flow

- **Final draft compilation**: Creates publication-ready output

The results speak for themselves. What would have taken hours of manual coordination across multiple tools is now handled automatically, with each step building logically on the previous one.

**Multi-modal workflow capabilities:**

- **Image generation and processing**: Integration with high-quality models including flux-schnell, flux-pro, and SDXL

- **Video generation and understanding**: Support for various stylized video models, including Seedance, Kling, and Veo

- **Audio generation tools**: Music generation through models like Lyria-2 and voice synthesis via models like Chatterbox

- **Integrated processing**: All multi-modal outputs can be referenced, analyzed, and reprocessed within the system


##  Conclusion

The integration of **Refly** and **Milvus** offers a pragmatic approach to automation—one that values reliability and ease of use over unnecessary complexity. By combining workflow orchestration with multi-modal processing, teams can move from concept to publication faster while retaining full control at every stage.

This isn’t about dismissing AI agents. They’re valuable for tackling genuinely complex, unpredictable problems. But for many automation needs—especially in content creation and data processing—a well-designed workflow can deliver better results with less overhead.

As AI tech evolves, the most effective systems will likely blend both strategies:

- **Workflows** where predictability, maintainability, and reproducibility are key.

- **Agents** where real reasoning, adaptability, and open-ended problem-solving are required.

The goal isn’t to build the flashiest AI—it’s to build the most _useful_ one. And often, the most helpful solution is also the most straightforward.


AI Agents or Workflows? Why Skip Agents for 80% of Automation


AI Agents or Workflows? Why You Should Skip Agents for 80% of Automation Tasks




**After months of speculation, OpenAI has finally shipped** [**GPT-5**](https://openai.com/gpt-5/)**.** The model isn’t the creative lightning strike that GPT-4 was, but for developers, especially those building agents and RAG pipelines, this release may quietly be the most useful upgrade yet. 

**TL;DR for builders:** GPT-5 unifies architectures, supercharges multimodal I/O, slashes factual error rates, extends context to 400k tokens, and makes large-scale usage affordable. However, the creativity and literary flair have taken a noticeable step back.


## What’s New Under the Hood? 

- **Unified core** — Merges GPT digital series with o-series reasoning models, delivering long-chain reasoning plus multimodal in a single architecture.

- **Full-spectrum multimodal** — Input/output across text, image, audio, and video, all within the same model.

- **Massive accuracy gains**: 

  - `gpt-5-main`: 44% fewer factual errors vs. GPT-4o.

  - `gpt-5-thinking`: 78% fewer factual errors vs. o3.

- **Domain skill boosts** — Stronger in code generation, mathematical reasoning, health consultation, and structured writing; hallucinations reduced significantly.

Alongside GPT-5, OpenAI also released **three additional variants**, each optimized for different needs:

![](https://assets.zilliz.com/gpt_5_family_99a9bee18a.png)

| **Model**  | **Description**                                       | **Input / $ per 1M tokens** | **Output / $ per 1M tokens** | **Knowledge Update** |
| ---------- | ----------------------------------------------------- | --------------------------- | ---------------------------- | -------------------- |
| gpt-5      | Main model, long-chain reasoning + full multimodal    | $1.25                       | $10.00                       | 2024-10-01           |
| gpt-5-chat | Equivalent to gpt-5, used in ChatGPT conversations    | —                           | —                            | 2024-10-01           |
| gpt-5-mini | 60% cheaper, retains ~90% of programming performance | $0.25                       | $2.00                        | 2024-05-31           |
| gpt-5-nano | Edge/offline, 32K context, latency <40ms              | $0.05                       | $0.40                        | 2024-05-31           |

GPT-5 broke records across 25 benchmark categories — from code repair to multimodal reasoning to medical tasks — with consistent accuracy improvements.

![](https://assets.zilliz.com/benchmark_1_8ebf1bed4b.png)

![](https://assets.zilliz.com/benchmark_2_c43781126f.png)


## Why Developers Should Care — Especially for RAG & Agents

Our hands-on tests suggest this release is a quiet revolution for Retrieval-Augmented Generation and agent-driven workflows.

1. **Price cuts** make experimentation viable — API input cost: **$1.25 per million tokens**; output cost: **$10**.

2. **A 400k context window** (vs. 128k in o3/4o) allows you to maintain state across complex multi-step agent workflows without context chopping.

3. **Fewer hallucinations & better tool use** — Supports multi-step chained tool calls, handles complex non-standard tasks, and improves execution reliability.

![](https://assets.zilliz.com/gpt_tool_call_e6a86fc59c.png)


## Not Without Flaws

Despite its technical advances, GPT-5 still shows clear limits. 

At launch, OpenAI’s keynote featured a slide that bizarrely calculated _52.8 > 69.1 = 30.8_, and in our own tests, the model confidently repeated the textbook-but-wrong “Bernoulli effect” explanation for airplane lift—reminding us **it’s still a pattern learner, not a true domain expert.**

**While STEM performance has sharpened, creative depth has slipped.** Many long-time users note a decline in literary flair: poetry feels flatter, philosophical conversations less nuanced, and long-form narratives more mechanical. The trade-off is clear—higher factual accuracy and stronger reasoning in technical domains, but at the expense of the artful, exploratory tone that once made GPT feel almost human.

With that in mind, let’s see how GPT-5 actually performs in our hands-on tests.


## Coding Tests

I started with a simple task: write an HTML script that allows users to upload an image and move it with the mouse. GPT-5 paused for about nine seconds, then produced working code that handled the interaction well. It felt like a good start.

![](https://assets.zilliz.com/gpt52_7b04c9b41b.gif)

The second task was harder: implement polygon–ball collision detection inside a rotating hexagon, with adjustable rotation speed, elasticity, and ball count. GPT-5 generated the first version in around thirteen seconds. The code included all expected features, but it had bugs and wouldn’t run.

![](https://assets.zilliz.com/image_6_3e6a34572a.png)

I then used the editor’s **Fix bug** option, and GPT-5 corrected the errors so the hexagon rendered. However, the balls never appeared — the spawn logic was missing or incorrect, meaning the core function of the program was absent despite the otherwise complete setup.

![](https://assets.zilliz.com/gpt51_6489df9914.gif)

**In summary,** GPT-5 can produce clean, well-structured interactive code and recover from simple runtime errors. But in complex scenarios, it still risks omitting essential logic, so human review and iteration are necessary before deployment.


## Reasoning Test

I posed a multi-step logic puzzle involving item colors, prices, and positional clues—something that would take most humans several minutes to solve.

![](https://assets.zilliz.com/reasoning_test_7ea15ed25b.png)

**Question:** _What is the blue item and what is its price?_

GPT-5 delivered the correct answer in just 9 seconds, with a clear and logically sound explanation. This test reinforced the model’s strength in structured reasoning and rapid deduction.


## Writing Test

I often turn to ChatGPT for help with blogs, social media posts, and other written content, so text generation is one of the capabilities I care about most. For this test, I asked GPT-5 to create a LinkedIn post based on a blog about Milvus 2.6’s multilingual analyzer.

![](https://assets.zilliz.com/Writing_Test_4fe5fef775.png)

The output was well-organized and hit all the key points from the original blog, but it felt too formal and predictable—more like a corporate press release than something meant to spark interest on a social feed. It lacked the warmth, rhythm, and personality that make a post feel human and inviting.

On the upside, the accompanying illustrations were excellent: clear, on-brand, and perfectly aligned with Zilliz’s tech style. Visually, it was spot-on; the writing just needs a bit more creative energy to match.


## Longer Context Window = Death of RAG and VectorDB?

We tackled this topic last year when [Google launched **Gemini 1.5 Pro**](https://zilliz.com/blog/will-retrieval-augmented-generation-RAG-be-killed-by-long-context-LLMs) with its ultra-long 10M-token context window. At the time, some people were quick to predict the end of RAG—and even the end of databases altogether. Fast-forward to today: not only is RAG still alive, it’s thriving. In practice, it’s become _more_ capable and productive, along with vector databases like [**Milvus**](https://milvus.io/) and [**Zilliz Cloud**](https://zilliz.com/cloud).

Now, with GPT-5’s expanded context length and more advanced tool-calling capabilities, the question has popped up again: _Do we still need vector databases for context ingestion, or even dedicated agents/RAG pipelines?_

**The short answer: absolutely yes. We still need them.**

Longer context is useful, but it’s not a replacement for structured retrieval. Multi-agent systems are still on track to be a long-term architectural trend—and these systems often need virtually unlimited context. Plus, when it comes to managing private, unstructured data securely, a vector database will always be the final gatekeeper.


## Conclusion

After watching OpenAI’s launch event and running my own hands-on tests, GPT-5 feels less like a dramatic leap forward and more like a refined blend of past strengths with a few well-placed upgrades. That’s not a bad thing—it’s a sign of the architectural and data-quality limits large models are starting to encounter.

As the saying goes, _severe criticism comes from high expectations_. Any disappointment around GPT-5 mostly comes from the very high bar OpenAI set for itself. And really—better accuracy, lower prices, and integrated multimodal support are still valuable wins. For developers building agents and RAG pipelines, this may actually be the most useful upgrade so far.

Some friends have been joking about making “online memorials” for GPT-4o, claiming their old chat companion’s personality is gone forever. I don’t mind the change—GPT-5 might be less warm and chatty, but its direct, no-nonsense style feels refreshingly straightforward.

**What about you?** Share your thoughts with us—join our [Discord](https://discord.com/invite/8uyFbECzPX), or join the conversation on [LinkedIn](https://www.linkedin.com/company/the-milvus-project/) and [X](https://x.com/milvusio). 


GPT-5 Review: Accuracy Up, Prices Down, Code Strong, Bad Creativity


GPT-5 Review: Accuracy Up, Prices Down, Code Strong — But Bad for Creativity



The AI world has been running hot. In just a few weeks, Anthropic dropped Claude 4.1 Opus, DeepMind stunned everyone with Genie 3 world simulator—and now, OpenAI steals the spotlight by open-sourcing two reasoning models: [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) and [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b), permissively licensed under Apache 2.0.

After launch, these models instantly shot to the #1 trending spot on Hugging Face—and for good reason. This is the first time since 2019 that OpenAI has released open-weight models that are actually production-ready. The move isn’t accidental—after years of pushing API-only access, OpenAI is clearly responding to the pressure from open-source leaders like DeepSeek, Meta’s LLaMA, and Qwen, who’ve been dominating both benchmarks and developer workflows.

In this post, we'll explore what makes GPT-oss different, how it compares to leading open models like DeepSeek R1 and Qwen 3, and why developers should care. We'll also walk through building a reasoning-capable RAG system using GPT-oss and Milvus, the most popular open-source vector database.  


## What Makes GPT-oss Special and Why You Should Care? 

GPT-oss isn't just another weight drop. It delivers in five key areas that matter to developers:


### 1: Built for Edge Deployment

GPT-oss comes in two strategically sized variants:

- gpt-oss-120b: 117B total, 5.1B active per token

- gpt-oss-20b: 21B total, 3.6B active per token

Using Mixture-of-Experts (MoE) architecture, only a subset of parameters is active during inference. This makes both models lightweight to run relative to their size:

- gpt-oss-120b runs on a single 80GB GPU (H100)

- gpt-oss-20b fits in just 16GB VRAM, meaning it runs on high-end laptops or edge devices

According to OpenAI's tests, gpt-oss-20b is the fastest OpenAI model for inference—ideal for low-latency deployments or offline reasoning agents.


### 2: Strong Benchmark Performance

According to OpenAI's evaluations:

- **gpt-oss-120b** performs near-parity with o4-mini on reasoning, tool use, and competition coding (Codeforces, MMLU, TauBench)

- **gpt-oss-20b** competes with o3-mini, and even outperforms it in math and healthcare reasoning


### 3: Cost-Efficient Training

OpenAI claims performance equivalent to o3-mini and o4-mini, but with dramatically lower training costs:

- **gpt-oss-120b**: 2.1 million H100-hours → ~$10M

- **gpt-oss-20b**: 210K H100-hours → ~$1M

Compare that to the multi-hundred-million-dollar budgets behind models like GPT-4. GPT-oss proves that efficient scaling and architecture choices can deliver competitive performance without a massive carbon footprint.


### 4: True Open-Source Freedom

GPT-oss uses Apache 2.0 licensing, which means:

- Commercial use allowed

- Full modification and redistribution rights

- No usage restrictions or copyleft clauses

This is really open source, not a research-only release. You can fine-tune for domain-specific use, deploy in production with full control, and build commercial products around it. Key features include configurable reasoning depth (low/medium/high), full chain-of-thought visibility, and native tool calling with structured output support.


### 5: Potential GPT-5 Preview

OpenAI hasn’t disclosed everything—but architecture details suggest this may preview the direction of **GPT-5**:

- Uses MoE with 4 experts per input

- Follows alternating dense + local sparse attention (GPT-3 pattern)

- Features more attention heads

- Interestingly, bias units from GPT-2 have made a comeback

If you're watching for signals on what comes next, GPT-oss may be the clearest public hint yet.


### Core Specifications

|              |                  |                   |             |                    |              |
| ------------ | ---------------- | ----------------- | ----------- | ------------------ | ------------ |
| **Model**    | **Total Params** | **Active Params** | **Experts** | **Context Length** | **VRAM Req** |
| gpt-oss-120b | 117B             | 5.1B              | 128         | 128k               | 80GB         |
| gpt-oss-20b  | 21B              | 3.6B              | 32          | 128k               | 16GB         |

Both models use the o200k_harmony tokenizer and support 128,000-token context length (roughly 96,000-100,000 words).


## GPT-oss vs. Other Reasoning Models

Here's how GPT-oss stacks up against OpenAI's internal models and top open-source competitors:

|                   |                         |             |                                      |
| ----------------- | ----------------------- | ----------- | ------------------------------------ |
| **Model**         | **Parameters (Active)** | **Memory**  | **Strengths**                        |
| **gpt-oss-120b**  | 117B (5.1B active)      | 80GB        | Single-GPU, open reasoning           |
| **gpt-oss-20b**   | 21B (3.6B active)       | 16GB        | Edge deployment, fast inference      |
| **DeepSeek R1**   | 671B (~37B active)     | Distributed | Benchmark leader, proven performance |
| **o4-mini (API)** | Proprietary             | API only    | Strong reasoning (closed)            |
| **o3-mini (API)** | Proprietary             | API only    | Lightweight reasoning (closed)       |

Based on various benchmarking models, here is what we found: 

- **GPT-oss vs. OpenAI's Own Models:** gpt-oss-120b matches o4-mini on competition math (AIME), coding (Codeforces), and tool use (TauBench). The 20b model performs similarly to o3-mini despite being much smaller.

- **GPT-oss vs. DeepSeek R1:** DeepSeek R1 dominates in pure performance but requires distributed infrastructure. GPT-oss offers simpler deployment—no distributed setup needed for the 120b model.

In summary, GPT-oss offers the best combination of performance, open access, and deployability. DeepSeek R1 wins on pure performance, but GPT-oss strikes the optimal balance for most developers. 


## Hands-on: Building with GPT-oss + Milvus

Now that we’ve seen what GPT-oss brings to the table, it’s time to put it to use. 

In the following sections, we'll walk through a hands-on tutorial for building a reasoning-capable RAG system using gpt-oss-20b and Milvus, all running locally, no API key required.


### Environment Setup

```
! pip install --upgrade "pymilvus[model]" openai requests tqdm
```

### Dataset Preparation

We'll use Milvus documentation as our knowledge base:

```
# Download and prepare Milvus docs
! wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
! unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs

from glob import glob
text_lines = []
for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()
    text_lines += file_text.split("# ")
```

### Model Setup

Access GPT-oss through [OpenRouter](https://openrouter.ai/openai/gpt-oss-20b:free) (or run locally). [**OpenRouter**](https://openrouter.ai/openai/gpt-oss-20b:free) is a platform that lets developers access and switch between multiple AI models (like GPT-4, Claude, Mistral) through a single, unified API. It's useful for comparing models or building apps that work with different AI providers. Now GPT-oss series have been available on OpenRouter now. 

![](https://assets.zilliz.com/1_46b575811f.png)

```
from openai import OpenAI

# Using OpenRouter for cloud access
openai_client = OpenAI(
    api_key="<OPENROUTER_API_KEY>",
    base_url="https://openrouter.ai/api/v1",
)

# Set up embedding model
from pymilvus import model as milvus_model
embedding_model = milvus_model.DefaultEmbeddingFunction()

# Test embedding dimensions
test_embedding = embedding_model.encode_queries(["This is a test"])[0]
embedding_dim = len(test_embedding)
print(embedding_dim)
print(test_embedding[:10])
```

```
768
[-0.04836066  0.07163023 -0.01130064 -0.03789345 -0.03320649 -0.01318448
 -0.03041712 -0.02269499 -0.02317863 -0.00426028]
```

### Set up Milvus vector database 
```
from pymilvus import MilvusClient

# Initialize Milvus client
milvus_client = MilvusClient(uri="http://localhost:19530", token="root:Milvus")
collection_name = "gpt_oss_rag_collection"

# Clean up existing collection
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)

# Create new collection
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim,
    metric_type="IP",  # Inner product distance
    consistency_level="Strong",
)
```

About MilvusClient parameter settings:

- Setting the URI to a local file (e.g., `./milvus.db`) is the most convenient method as it automatically uses Milvus Lite to store all data in that file.

- For large-scale data, you can set up a more powerful Milvus server on Docker or Kubernetes. In this case, use the server’s URI (e.g., `http://localhost:19530`) as your URI.

- If you want to use [Zilliz Cloud ](https://zilliz.com/cloud)(the managed service of Milvus), adjust the URI and token, which correspond to the Public Endpoint and API key in Zilliz Cloud.


### Adding Documents to the Collection

Now we’ll create embeddings for our text chunks and add them to Milvus:

```
from tqdm import tqdm

data = []
doc_embeddings = embedding_model.encode_documents(text_lines)

for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": doc_embeddings[i], "text": line})

milvus_client.insert(collection_name=collection_name, data=data)
```

Output:

```
Creating embeddings: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 72/72 [00:00<00:00, 1222631.13it/s]
{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
```

### RAG Query Pipeline

Now for the exciting part - let’s set up our RAG system to answer questions.

Let’s specify a common question about Milvus:
```
question = "How is data stored in milvus?"
```

Search for this question in the collection and retrieve the top 3 semantically matching results:

```
search_res = milvus_client.search(
    collection_name=collection_name,
    data=embedding_model.encode_queries(
        [question]
    ),  # Convert the question to an embedding vector
    limit=3,  # Return top 3 results
    search_params={"metric_type": "IP", "params": {}},  # Inner product distance
    output_fields=["text"],  # Return the text field
)
```

Let’s look at the search results for this query:
```
import json

retrieved_lines_with_distances = [
    (res["entity"]["text"], res["distance"]) for res in search_res[0]
]
print(json.dumps(retrieved_lines_with_distances, indent=4))
```

```
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.6572664976119995
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.6312144994735718
    ],
    [
        "How does Milvus handle vector data types and precision?\n\nMilvus supports Binary, Float32, Float16, and BFloat16 vector types.\n\n- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.\n- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.\n- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.\n\n###",
        0.6115782856941223
    ]
]
```

### Using the GPT-oss to Build a RAG Response

Convert the retrieved documents to string format:

```
context = "\n".join(
    [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
)
```
Provide system prompt and user prompt for the large language model:
```
SYSTEM_PROMPT = """
Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
"""
USER_PROMPT = f"""
Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
<context>
{context}
</context>
<question>
{question}
</question>
"""
```
Use the latest gpt-oss model to generate a response based on the prompt:
```
response = openai_client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
)
print(response.choices[0].message.content)
```

```
Milvus stores its data in two distinct layers:

| Type of data | Where it is stored | How it is stored |
|-------------|-------------------|-----------------|
| **Inserted data** (vector data, scalar fields, collection‑specific schema) | In the **persistent object storage** configured for the cluster. The data are written as **incremental logs** (append‑only logs) that are persisted by the DataNode. | The DataNode reads from the message‑queue and writes the incoming data into the storage backend (MinIO, AWS S3, GCS, Azure Blob, Alibaba OSS, Tencent COS, etc.). When a `flush()` call is issued, the DataNode forces all queued data to be written to the persistent storage immediately. |
| **Metadata** (information about collections, partitions, indexes, etc.) | In **etcd**. Each Milvus module (catalog, index, etc.) keeps its own metadata. | The metadata is generated and managed by Milvus and persisted in the distributed key‑value store **etcd**. |

**Summary:**  
- **Inserted data** = incremental logs stored in the chosen object‑storage backend.  
- **Metadata** = stored in the distributed configuration store **etcd**.  


Together, these two storage mechanisms (object storage for the actual data and etcd for metadata) make up Milvus’s data‑storage architecture.
```

## Final Thoughts on GPT-oss

GPT-oss is OpenAI’s quiet admission that open-source can't be ignored anymore. It doesn’t blow DeepSeek R1 or Qwen 3 or many other models out of the water, but it does bring something they don’t: OpenAI’s training pipeline, applied to a model you can actually inspect and run locally.

**Performance? Solid. Not mind-blowing, but dependable.** The 20B model running on consumer hardware—or even mobile with LM Studio—is the kind of practical advantage that actually matters to developers. It’s more “this just works” than “wow, this changes everything.” And honestly, that’s fine.

**Where it falls short is multilingual support.** If you’re working in anything other than English, you’ll hit weird phrasing, spelling issues, and general confusion. The model was clearly trained with an English-first lens. If global coverage matters, you’re probably going to need to fine-tune it with a multilingual dataset. 

What’s most interesting, though, is the timing. OpenAI’s teaser on X—with a “5” dropped into the word “LIVESTREAM”—feels like a setup. GPT-oss might not be the main act, but it could be a preview of what’s coming in GPT-5. Same ingredients, different scale. Let’s wait.

![](https://assets.zilliz.com/2_0fed950b8e.png)

**The real win is having more high-quality choices.** Competition drives innovation, and OpenAI re-entering open-source development benefits everyone. Test GPT-oss against your specific requirements, but choose based on what actually works for your use case, not brand recognition.


GPT-oss vs o4-mini: Edge-Ready, Solid, But Not Disruptive


GPT-oss vs o4-mini: Edge-Ready, On-Par Performance — Dependable, Not Mind-Blowing


Deploy Milvus on K8s Just Got Easier with the Milvus Operator 


Deploying Milvus on Kubernetes Just Got Easier with the Milvus Operator




Ever wished you could just tell your AI assistant, _"Show me all collections in my vector database"_ or _"Find documents similar to this text"_ and have it actually work?

The [**Milvus MCP Server**](http://github.com/zilliztech/mcp-server-milvus) makes this possible by connecting your Milvus vector database directly to AI coding assistants like Claude Desktop and Cursor IDE through Model Context Protocol (MCP). Instead of writing `pymilvus` code, you can manage your entire Milvus through natural language conversations.

- Without Milvus MCP Server: Writing Python scripts with pymilvus SDK to search vectors

- With Milvus MCP Server: "Find documents similar to this text in my collection."

👉 **GitHub Repository:**[ github.com/zilliztech/mcp-server-milvus](https://github.com/zilliztech/mcp-server-milvus)

And if you're using [Zilliz Cloud](https://zilliz.com/cloud) (managed Milvus), we’ve got you covered too. At the end of this blog, we’ll also introduce the **Zilliz MCP Server**, a managed option that works seamlessly with Zilliz Cloud. Let’s dive in.


## What You'll Get with Milvus MCP Server

The Milvus MCP Server gives your AI assistant the following capabilities:

- **List and explore** vector collections

- **Search vectors** using semantic similarity

- **Create new collections** with custom schemas

- **Insert and manage** vector data

- **Run complex queries** without writing code

- And more 

All through natural conversation, as if you're talking to a database expert. Check out [this repo](https://github.com/zilliztech/mcp-server-milvus?tab=readme-ov-file#available-tools) for the complete list of capabilities. 

![](https://assets.zilliz.com/demo_adedb25430.gif)


## Quick Start Guide

### Prerequisites

**Required:**

- Python 3.10 or higher

- A running Milvus instance (local or remote)

- [uv package manager](https://github.com/astral-sh/uv) (recommended)

**Supported AI Applications:**

- Claude Desktop

- Cursor IDE

- Any MCP-compatible application


### Tech Stack We’ll Use 

In this tutorial, we'll use the following tech stack:

- **Language Runtime:** [Python 3.11](https://www.python.org/)

- **Package Manager:** UV

- **IDE:** Cursor

- **MCP Server:** mcp-server-milvus

- **LLM:** Claude 3.7

- **Vector Database:** Milvus


### Step 1: Install Dependencies

First, install the uv package manager:

```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Or:

```
pip3 install uv -i https://mirrors.aliyun.com/pypi/simple
```

Verify installation:

```
uv --version
uvx --version
```


![](https://assets.zilliz.com/Step_1_Install_Dependencies_3e452c55e3.png)


### Step 2: Set Up Milvus

[Milvus](https://milvus.io/) is an open-source vector database native for AI workloads, created by [Zilliz](https://zilliz.com/). Designed to handle millions to billions of vector records, it has gained over 36,000 stars on GitHub. Building on this foundation, Zilliz also offers [Zilliz Cloud](https://zilliz.com/cloud)—a fully managed service of Milvus engineered for usability, cost-efficiency, and security with a cloud-native architecture.

For Milvus deployment requirements, visit [this guide on the doc site](https://milvus.io/docs/prerequisite-docker.md).  

**Minimum requirements:**

- **Software:** Docker, Docker Compose

- **RAM:** 16GB+

- **Disk:** 100GB+

Download the deployment YAML file:

```
[root@Milvus ~]# wget https://github.com/milvus-io/milvus/releases/download/v2.5.4/milvus-standalone-docker-compose.yml -O docker-compose.yml
```

Start Milvus:

```
[root@Milvus ~]# docker-compose up -d
```

```
[root@Milvus ~]# docker ps -a
```



![](https://assets.zilliz.com/Step_2_Set_Up_Milvus_4826468767.png)

Your Milvus instance will be available at `http://localhost:19530`.


### Step 3: Install the MCP Server

Clone and test the MCP server:

```
git clone https://github.com/zilliztech/mcp-server-milvus.git
cd mcp-server-milvus

# Test the server locally
uv run src/mcp_server_milvus/server.py --milvus-uri http://localhost:19530
```

We recommend installing dependencies and verifying locally before registering the server in Cursor:

```
uv run src/mcp_server_milvus/server.py --milvus-uri http://192.168.4.48:19530
```


If you see the server start successfully, you're ready to configure your AI tool.

![](https://assets.zilliz.com/Step_3_Install_the_MCP_Server_9ce01351e6.png)


### Step 4: Configure Your AI Assistant

**Option A: Claude Desktop**

1. #### Install Claude Desktop from `[claude.ai/download](http://claude.ai/download)`.

2. Open the configuration file:

- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Windows: `%APPDATA%\Claude\claude_desktop_config.json`

Add this configuration:

```
{
  "mcpServers": {
    "milvus": {
      "command": "/path/to/uv",
      "args": [
        "--directory",
        "/path/to/mcp-server-milvus",
        "run",
        "src/mcp_server_milvus/server.py",
        "--milvus-uri",
        "http://localhost:19530"
      ]
    }
  }
}
```



4. Restart Claude Desktop

**Option B: Cursor IDE**

1. Open Cursor Settings → Features → MCP

2. Add new global MCP server (this creates `.cursor/mcp.json`)

3. Add this configuration:

Note: Adjust paths to your actual file structure.

```
{
  "mcpServers": {
    "milvus": {
      "command": "/PATH/TO/uv",
      "args": [
        "--directory",
        "/path/to/mcp-server-milvus/src/mcp_server_milvus",
        "run",
        "server.py",
        "--milvus-uri",
        "http://127.0.0.1:19530"
      ]
    }
  }
}
```


![](https://assets.zilliz.com/Option_B_Cursor_IDE_cd1321ea25.png)

**Parameters:**

- `/PATH/TO/uv` is the path to the uv executable
- `--directory` is the path to the cloned project
- `--milvus-uri` is your Milvus server endpoint

4. Restart the Cursor or reload the window

**Pro tip:** Find your `uv` path with `which uv` on macOS/Linux or `where uv`  on Windows.


### Step 5: See It in Action

Once configured, try these natural language commands:

- **Explore your database:** "What collections do I have in my Milvus database?"

- **Create a new collection:** "Create a collection called 'articles' with fields for title (string), content (string), and a 768-dimension vector field for embeddings."

- **Search for similar content:** "Find the five most similar articles to 'machine learning applications' in my articles collection."

- **Insert data:** "Add a new article with title 'AI Trends 2024' and content 'Artificial intelligence continues to evolve...' to the articles collection"

**What used to require 30+ minutes of coding now takes seconds of conversation.**

You get real-time control and natural language access to Milvus—without writing boilerplate or learning the API.


## Troubleshooting

If MCP tools don't appear, restart your AI application completely, verify the UV path with `which uv`, and test the server manually with `uv run src/mcp_server_milvus/server.py --milvus-uri http://localhost:19530`.

For connection errors, check that Milvus is running with `docker ps | grep milvus`, try using `127.0.0.1` instead of `localhost`, and verify port 19530 is accessible.

If you encounter authentication issues, set the `MILVUS_TOKEN` environment variable if your Milvus requires authentication, and verify your permissions for the operations you're attempting.


## Managed Alternative: Zilliz MCP Server

The open-source **Milvus MCP Server** is a great solution for local or self-hosted deployments of Milvus. But if you’re using [Zilliz Cloud](https://zilliz.com/cloud)—the fully managed, enterprise-grade service built by the creators of Milvus—there’s a purpose-built alternative: the [**Zilliz MCP Server**](https://zilliz.com/blog/introducing-zilliz-mcp-server).

[Zilliz Cloud](https://zilliz.com/cloud) eliminates the overhead of managing your own Milvus instance by offering a scalable, performant, and secure cloud-native vector database. The **Zilliz MCP Server** integrates directly with Zilliz Cloud and exposes its capabilities as MCP-compatible tools. This means your AI assistant—whether in Claude, Cursor, or another MCP-aware environment—can now query, manage, and orchestrate your Zilliz Cloud workspace using natural language.

![](https://assets.zilliz.com/zilliz_mcp_abe1ca1271.png)

No boilerplate code. No switching tabs. No manually writing REST or SDK calls. Just say your request and let your assistant handle the rest.


### 🚀 Getting Started with Zilliz MCP Server

If you're ready for production-ready vector infrastructure with the ease of natural language, getting started takes just a few steps:

1. [**Sign up for Zilliz Cloud**](https://cloud.zilliz.com/signup) – free tier available.

2. [**Install the Zilliz MCP Server** ](http://github.com/zilliztech/zilliz-mcp-server)from the GitHub repository.

3. **Configure your MCP-compatible assistant** (Claude, Cursor, etc.) to connect to your Zilliz Cloud instance.

This gives you the best of both worlds: powerful vector search with production-grade infrastructure, now accessible through plain English.


## Wrapping Up

And that’s it—you’ve just learned how to turn Milvus into a natural language-friendly vector database you can literally _talk to_. No more digging through SDK docs or writing boilerplate just to spin up a collection or run a search.

Whether you’re running Milvus locally or using Zilliz Cloud, the MCP Server gives your AI assistant a toolbox to manage your vector data like a pro. Just type what you want to do, and let Claude or Cursor handle the rest.

So go ahead—fire up your AI dev tool, ask “what collections do I have?” and see it in action. You’ll never want to go back to writing vector queries by hand.

- Local setup? Use the open-source[ Milvus MCP Server](https://github.com/zilliztech/mcp-server-milvus)

- Prefer a managed service? Sign up for Zilliz Cloud and use the[ Zilliz MCP Server](https://github.com/zilliztech/zilliz-mcp-server)

You’ve got the tools. Now let your AI do the typing. 


Talk to Your Vector Database: Managing Milvus via Natural Language



## Introduction 

Modern AI applications are becoming increasingly complex. You can't just throw one search method at a problem and call it done.

Take recommendation systems, for example—they require **vector search** to understand the meaning of text and images, **metadata filtering** to narrow results by price, category, or location, and **keyword search** for direct queries like "Nike Air Max." Each method solves a different part of the problem, and real-world systems need all of them working together.

The future of search isn’t about choosing between vector and keyword. It’s about combining vector AND keyword AND filtering, along with other search types—all in one place. That’s why we started building [hybrid search](https://milvus.io/docs/hybrid_search_with_milvus.md) into Milvus a year ago, with the release of Milvus 2.5.


## But Full-Text Search Works Differently

Bringing full-text search into a vector-native system isn’t easy. Full-text search has its own set of challenges.

While vector search captures the _semantic_ meaning of text—turning it into high-dimensional vectors—full-text search depends on understanding **the structure of language**: how words are formed, where they begin and end, and how they relate to one another. For instance, when a user searches for "running shoes" in English, the text goes through several processing steps: 

_Split on whitespace → lowercase → remove stopwords → stem "running" to "run"._

To handle this correctly, we need a robust **language analyzer**—one that handles splitting, stemming, filtering, and more.

When we introduced [BM25 full-text search](https://milvus.io/docs/full-text-search.md#Full-Text-Search) in Milvus 2.5, we included a customizable analyzer, and it worked well for what it was designed to do. You could define a pipeline using tokenizers, token filters, and character filters to prepare text for indexing and search.

For English, this setup was relatively straightforward. But things become more complex when you deal with multiple languages.


## The Challenge of Multilingual Full-Text Search

Multilingual full-text search introduces a range of challenges:

- **Complex languages need special treatment**: Languages like Chinese, Japanese, and Korean don’t use spaces between words. They need advanced tokenizers to segment characters into meaningful words. These tools may work well for a single language but rarely support multiple complex languages simultaneously.

- **Even similar languages can conflict**: English and French might both use whitespace to separate words, but once you apply language-specific processing like stemming or lemmatization, one language’s rules can interfere with the other’s. What improves accuracy for English might distort French queries—and vice versa.

In short, **different languages require different analyzers**. Trying to process Chinese text with an English analyzer leads to failure—there are no spaces to split on, and English stemming rules can corrupt Chinese characters.

The bottom line? Relying on a single tokenizer and analyzer for multilingual datasets makes it nearly impossible to ensure consistent, high-quality tokenization across all languages. And that leads directly to degraded search performance.

As teams began adopting full-text search in Milvus 2.5, we started hearing the same feedback:

_"This is perfect for our searches in English, but what about our multilingual customer support tickets?" "We love having both vector and BM25 search, but our dataset includes Chinese, Japanese, and English content." "Can we get the same search precision across all our languages?"_

These questions confirmed what we had already seen in practice: full-text search fundamentally differs from vector search. Semantic similarity works well across languages, but accurate text search requires a deep understanding of each language’s structure.

That’s why [Milvus 2.6](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md) introduces a completely overhauled text analysis pipeline with comprehensive multi-language support. This new system automatically applies the correct analyzer for each language, enabling accurate and scalable full-text search across multilingual datasets, without manual configuration or compromise in quality.




## How Milvus 2.6 Enables Robust Multilingual Full-Text Search

After extensive research and development, we've built a suite of features that address different multilingual scenarios. Each approach solves the language-dependency problem in its own way.


### 1. Multi-Language Analyzer: Precision Through Control

The [**Multi-Language Analyzer**](https://milvus.io/docs/multi-language-analyzers.md#Multi-language-Analyzers) allows you to define different text processing rules for different languages within the same collection, instead of forcing all languages through the same analysis pipeline. 

**Here's how it works:** you configure language-specific analyzers and tag each document with its language during insertion. When performing a BM25 search, you specify which language analyzer to use for query processing. This ensures that both your indexed content and search queries are processed with the optimal rules for their respective languages.

**Perfect for:** Applications where you know the language of your content and want maximum search precision. Think multinational knowledge bases, localized product catalogs, or region-specific content management systems.

**The requirement:** You need to provide language metadata for each document. Currently only available for BM25 search operations.


### 2. Language Identifier Tokenizer: Automatic Language Detection

We know that manually tagging every piece of content isn't always practical. The [**Language Identifier Tokenizer**](https://milvus.io/docs/multi-language-analyzers.md#Overview) brings automatic language detection directly into the text analysis pipeline.

**Here's how it works:** This intelligent tokenizer analyzes incoming text, detects its language using sophisticated detection algorithms, and automatically applies the appropriate language-specific processing rules. You configure it with multiple analyzer definitions - one for each language you want to support, plus a default fallback analyzer.

We support two detection engines: `whatlang` for faster processing and `lingua` for higher accuracy. The system supports 71-75 languages, depending on your chosen detector. During both indexing and search, the tokenizer automatically selects the right analyzer based on detected language, falling back to your default configuration when detection is uncertain.

**Perfect for:** Dynamic environments with unpredictable language mixing, user-generated content platforms, or applications where manual language tagging isn't feasible.

**The trade-off:** Automatic detection adds processing latency and may struggle with very short text or mixed-language content. But for most real-world applications, the convenience significantly outweighs these limitations.


### 3. ICU Tokenizer: Universal Foundation

If the first two options feel like overkill, we've got something simpler for you. We've newly integrated the[ ICU (International Components for Unicode) tokenizer](https://milvus.io/docs/icu-tokenizer.md#ICU) into Milvus 2.6. ICU has been around forever - it's a mature, widely-used set of libraries that handles text processing for tons of languages and scripts. The cool thing is that it can handle various complex and simple languages all at once.

The ICU tokenizer is honestly a great default choice. It uses Unicode-standard rules for breaking up words, which makes it reliable for dozens of languages that don't have their own specialized tokenizers. If you just need something powerful and general-purpose that works well across multiple languages, ICU does the job.

**Limitation:** ICU still works within a single analyzer, so all your languages end up sharing the same filters. Want to do language-specific stuff like stemming or lemmatization? You'll run into the same conflicts we talked about earlier.

**Where it really shines:** We built ICU to work as the default analyzer within the multi-language or language identifier setups. It's basically your intelligent safety net for handling languages you haven't explicitly configured.


## See It in Action: Hands-On Demo

Enough theory—let’s dive into some code! Here’s how to use the new multilingual features in **pymilvus** to build a multilingual search collection.

We’ll start by defining some reusable analyzer configurations, then walk through **two complete examples**:

- Using the **Multi-Language Analyzer**

- Using the **Language Identifier Tokenizer**

👉 For the complete demo code, check out [this GitHub page](https://github.com/milvus-io/pymilvus/tree/master/examples/full_text_search).



### Step 1: Set up the Milvus Client

_First, we connect to Milvus, set a collection name, and clean up any existing collections to start fresh._

```
from pymilvus import MilvusClient, DataType, Function, FunctionType

# 1. Setup Milvus Client
client = MilvusClient("http://localhost:19530")
COLLECTION_NAME = "multilingual_test"
if client.has_collection(collection_name=COLLECTION_NAME):
    client.drop_collection(collection_name=COLLECTION_NAME)
```


### Step 2: Define Analyzers for Multiple Languages

Next, we define an `analyzers` dictionary with language-specific configurations. These will be used in both multilingual search methods shown later.

```
# 2. Define analyzers for multiple languages
# These individual analyzer definitions will be reused by both methods.
analyzers = {
    "Japanese": { 
        # Use lindera with japanese dict 'ipadic' 
        # and remove punctuation beacuse lindera tokenizer will remain punctuation
        "tokenizer":{
            "type": "lindera",
            "dict_kind": "ipadic"
        },
        "filter": ["removepunct"]
    },
    "English": {
        # Use build-in english analyzer
        "type": "english",
    },
    "default": {
        # use icu tokenizer as a fallback.
        "tokenizer": "icu",
    }
}
```


### Option A: Using The Multi-Language Analyzer

This approach is best when you **know the language of each document ahead of time**. You'll pass that information through a dedicated ` language` field during data insertion.


#### Create a Collection with Multi-Language Analyzer

We’ll create a collection where the ` "text"` field uses different analyzers depending on the `language` field value.

```
# --- Option A: Using Multi-Language Analyzer ---
print("\n--- Demonstrating Multi-Language Analyzer ---")

# 3A. reate a collection with the Multi Analyzer

mutil_analyzer_params = {
    "by_field": "language",
    "analyzers": analyzers,
}

schema = MilvusClient.create_schema(
    auto_id=True,
    enable_dynamic_field=False,
)
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)# Apply our multi-language analyzer to the 'title' field
schema.add_field(field_name="language", datatype=DataType.VARCHAR, max_length=255, nullable = True)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=25565, enable_analyzer=True, multi_analyzer_params = mutil_analyzer_params)
schema.add_field(field_name="text_sparse", datatype=DataType.SPARSE_FLOAT_VECTOR) # Bm25 Sparse Vector

# add bm25 function
text_bm25_function = Function(
    name="text_bm25",
    function_type=FunctionType.BM25,
    input_field_names=["text"],
    output_field_names=["text_sparse"],
)
schema.add_function(text_bm25_function)

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="text_sparse",
    index_type="AUTOINDEX", # Use auto index for BM25
    metric_type="BM25",
)

client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema,
    index_params=index_params
)
print(f"Collection '{COLLECTION_NAME}' created successfully.")
```


#### Insert Multilingual Data and Load Collection

Now insert documents in English and Japanese. The `language` field tells Milvus which analyzer to use.

```
# 4A. Insert data for Multi-Language Analyzer and load collection# Insert English and Japanese movie titles, explicitly setting the 'language' field
client.insert(
    collection_name=COLLECTION_NAME,
    data=[
        {"text": "The Lord of the Rings", "language": "English"},
        {"text": "Spirited Away", "language": "English"},
        {"text": "千と千尋の神隠し", "language": "Japanese"}, # This is "Spirited Away" in Japanese
        {"text": "君の名は。", "language": "Japanese"}, # This is "Your Name." in Japanese
    ]
)
print(f"Inserted multilingual data into '{COLLECTION_NAME}'.")

# Load the collection into memory before searching
client.load_collection(collection_name=COLLECTION_NAME)
```


#### Run Full-Text Search

To search, specify which analyzer to use for the query based on its language.

```
# 5A. Perform a full-text search with Multi-Language Analyzer# When searching, explicitly specify the analyzer to use for the query string.
print("\n--- Search results for Multi-Language Analyzer ---")
results_multi_jp = client.search(
    collection_name=COLLECTION_NAME,
    data=["神隠し"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25", "analyzer_name": "Japanese"}, # Specify Japanese analyzer for query
    consistency_level = "Strong",
)
print("\nSearch results for '神隠し' (Multi-Language Analyzer):")
for result in results_multi_jp[0]:
    print(result)

results_multi_en = client.search(
    collection_name=COLLECTION_NAME,
    data=["Rings"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25", "analyzer_name": "English"}, # Specify English analyzer for query
    consistency_level = "Strong",
)
print("\nSearch results for 'Rings' (Multi-Language Analyzer):")
for result in results_multi_en[0]:
    print(result)

client.drop_collection(collection_name=COLLECTION_NAME)
print(f"Collection '{COLLECTION_NAME}' dropped.")
```


#### Results: 

![](https://assets.zilliz.com/1_results_561f628de3.png)


### Option B: Using the Language Identifier Tokenizer

This approach takes the manual language handling out of your hands. The **Language Identifier Tokenizer** automatically detects the language of each document and applies the correct analyzer—no need to specify a `language` field.


#### Create a Collection with Language Identifier Tokenizer

Here, we create a collection where the `"text"` field uses automatic language detection to choose the right analyzer.

```
# --- Option B: Using Language Identifier Tokenizer ---
print("\n--- Demonstrating Language Identifier Tokenizer ---")

# 3A. create a collection with language identifier
analyzer_params_langid = {
    "tokenizer": {
        "type": "language_identifier",
        "analyzers": analyzers # Referencing the analyzers defined in Step 2
    },
}

schema_langid = MilvusClient.create_schema(
    auto_id=True,
    enable_dynamic_field=False,
)
schema_langid.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
# The 'language' field is not strictly needed by the analyzer itself here, as detection is automatic.# However, you might keep it for metadata purposes.
schema_langid.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=25565, enable_analyzer=True, analyzer_params = analyzer_params_langid)
schema_langid.add_field(field_name="text_sparse", datatype=DataType.SPARSE_FLOAT_VECTOR) # BM25 Sparse Vector# add bm25 function
text_bm25_function_langid = Function(
    name="text_bm25",
    function_type=FunctionType.BM25,
    input_field_names=["text"],
    output_field_names=["text_sparse"],
)
schema_langid.add_function(text_bm25_function_langid)

index_params_langid = client.prepare_index_params()
index_params_langid.add_index(
    field_name="text_sparse",
    index_type="AUTOINDEX", # Use auto index for BM25
    metric_type="BM25",
)

client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema_langid,
    index_params=index_params_langid
)
print(f"Collection '{COLLECTION_NAME}' created successfully with Language Identifier Tokenizer.")
```


#### Insert Data and Load Collection

Insert text in different languages—no need to label them. Milvus detects and applies the correct analyzer automatically.

```
# 4B. Insert Data for Language Identifier Tokenizer and Load Collection
# Insert English and Japanese movie titles. The language_identifier will detect the language.
client.insert(
    collection_name=COLLECTION_NAME,
    data=[
        {"text": "The Lord of the Rings"},
        {"text": "Spirited Away"},
        {"text": "千と千尋の神隠し"}, 
        {"text": "君の名は。"},
    ]
)
print(f"Inserted multilingual data into '{COLLECTION_NAME}'.")

# Load the collection into memory before searching
client.load_collection(collection_name=COLLECTION_NAME)
```


#### Run Full-Text Search

Here’s the best part: **no need to specify an analyzer** when searching. The tokenizer automatically detects the query language and applies the right logic.

```
# 5B. Perform a full-text search with Language Identifier Tokenizer# No need to specify analyzer_name in search_params; it's detected automatically for the query.
print("\n--- Search results for Language Identifier Tokenizer ---")
results_langid_jp = client.search(
    collection_name=COLLECTION_NAME,
    data=["神隠し"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25"}, # Analyzer automatically determined by language_identifier
    consistency_level = "Strong",
)
print("\nSearch results for '神隠し' (Language Identifier Tokenizer):")
for result in results_langid_jp[0]:
    print(result)

results_langid_en = client.search(
    collection_name=COLLECTION_NAME,
    data=["the Rings"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25"}, # Analyzer automatically determined by language_identifier
    consistency_level = "Strong",
)
print("\nSearch results for 'the Rings' (Language Identifier Tokenizer):")
for result in results_langid_en[0]:
    print(result)

client.drop_collection(collection_name=COLLECTION_NAME)
print(f"Collection '{COLLECTION_NAME}' dropped.")
```


#### Results

![](https://assets.zilliz.com/2_results_486712c3f6.png)


## Conclusion

Milvus 2.6 takes a big step forward in making **hybrid search** more powerful and accessible, combining vector search with keyword search, now across multiple languages. With the enhanced multilingual support, you can build apps that understand _what users mean_ and _what they say_, no matter what language they're using.

But that’s just one part of the update. Milvus 2.6 also brings several other features that make search faster, smarter, and easier to work with:

- **Better Query Matching** – Use `phrase_match` and `multi_match` for more accurate searches

- **Faster JSON Filtering** – Thanks to a new, dedicated index for JSON fields

- **Scalar-Based Sorting** – Sort results by any numeric field

- **Advanced Reranking** – Reorder results using models or custom scoring logic

Want the complete breakdown of Milvus 2.6? Check out our latest post: [**Introducing Milvus 2.6: Affordable Vector Search at Billion Scale**](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)**.** 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus).


How Milvus How Milvus 2.6 Upgrades Multilingual Full-Text Search at Scale


How Milvus 2.6 Upgrades Multilingual Full-Text Search at Scale



The AI coding assistant battlefield is heating up fast. We've seen [Claude Code](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md#Claude-Code-or-Gemini-CLI-Choose-Your-Co-Pilot-Wisely) from Anthropic making waves, Google's [Gemini CLI](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md#Claude-Code-or-Gemini-CLI-Choose-Your-Co-Pilot-Wisely) shaking up terminal workflows, OpenAI's Codex powering GitHub Copilot, Cursor winning over VS Code users, and **now Alibaba Cloud enters with Qwen Code.**

Honestly, this is great news for developers. More players mean better tools, innovative features, and most importantly, **open-source alternatives** to expensive proprietary solutions. Let's learn what this latest player brings to the table.


## Meet Qwen3-Coder and Qwen Code

Alibaba Cloud recently released[ **Qwen3-Coder**](https://github.com/QwenLM/Qwen3-Coder), an open-source agentic coding model achieving state-of-the-art results across multiple benchmarks. They also launched[ **Qwen Code**](https://github.com/QwenLM/qwen-code), an open-source AI coding CLI tool built on Gemini CLI but enhanced with specialized parsers for Qwen3-Coder. 

The flagship model, **Qwen3-Coder-480B-A35B-Instruct**, delivers impressive capabilities: native support for 358 programming languages, 256K token context window (expandable to 1M tokens via YaRN), and seamless integration with Claude Code, Cline, and other coding assistants.


## The Universal Blind Spot in Modern AI Coding Copilots

While Qwen3-Coder is powerful, I am more interested in its coding assistant: **Qwen Code**. Here's what I found interesting. Despite all the innovation, Qwen Code shares the exact same limitation as Claude Code and Gemini CLI: **_they're great at generating fresh code but struggle with understanding existing codebases._**

Take this example: you ask Gemini CLI or Qwen Code to "find where this project handles user authentication." The tool starts hunting for obvious keywords like "login" or "password" but completely misses that critical `verifyCredentials()` function. Unless you're willing to burn through tokens by feeding your entire codebase as context—which is both expensive and time-consuming—these tools hit a wall pretty quickly.

**_This is the real gap in today's AI tooling: intelligent code context understanding._**


## Supercharge Any Coding Copilot with Semantic Code Search

What if you could give any AI coding copilot—whether it's Claude Code, Gemini CLI, or Qwen Code—the ability to truly understand your codebase semantically? What if you could build something as powerful as Cursor for your own projects without the hefty subscription fees, while maintaining complete control over your code and data?

Well, enter[ **Code Context**](https://github.com/zilliztech/code-context)—an open-source, MCP-compatible plugin that transforms any AI coding agent into a context-aware powerhouse. It's like giving your AI assistant the institutional memory of a senior developer who's worked on your codebase for years. Whether you're using Qwen Code, Claude Code, Gemini CLI, working in VSCode, or even coding in Chrome, **Code Context** brings semantic code search to your workflow.

Ready to see how this works? Let's build an enterprise-grade AI coding copilot using **Qwen3-Coder + Qwen Code + Code Context**.


## Hands-On Tutorial: Building Your Own AI Coding Copilot 

### Prerequisites

Before we begin, ensure you have:

- **Node.js 20+** installed

- **OpenAI API key** ([Get one here](https://openai.com/index/openai-api/))

- **Alibaba Cloud account** for Qwen3-Coder access ([get one here](https://www.alibabacloud.com/en)) 

- **Zilliz Cloud account** for vector database ([Register here](https://cloud.zilliz.com/login) for free if you don’t have one yet)

**Notes: 1)** In this tutorial, we’ll use Qwen3-Coder-Plus, the commercial version of Qwen3-Coder, because of its strong coding capabilities and ease of use. If you prefer an open-source option, you can use qwen3-coder-480b-a35b-instruct instead. 2) While Qwen3-Coder-Plus offers excellent performance and usability, it comes with high token consumption. Be sure to factor this into your enterprise budgeting plans.


### Step 1: Environment Setup

Verify your Node.js installation:

```
curl -qL https://www.npmjs.com/install.sh | sh
```



### Step 2: Install Qwen Code

```
npm install -g @qwen-code/qwen-code
qwen --version
```


If you see the version number like below, it means the installation was successful.

![](https://assets.zilliz.com/1_0d5ebc152e.png)


### Step 3: Configure Qwen Code

Navigate to your project directory and initialize Qwen Code.

```
Qwen
```


Then, you’ll see a page like below. 

![](https://assets.zilliz.com/2_e6598ea982.png)

**API Configuration Requirements:**

- API Key: Obtain from[ Alibaba Cloud Model Studio](https://modelstudio.console.alibabacloud.com/)

- Base URL: `https://dashscope.aliyuncs.com/compatible-mode/v1`

- Model Selection:

  - `qwen3-coder-plus` (commercial version, most capable)

  - `qwen3-coder-480b-a35b-instruct` (open-source version)

![](https://assets.zilliz.com/3_5ed0c54084.png)

After configuration, press **Enter** to proceed.


### Step 4: Test Basic Functionality

Let's verify your setup with two practical tests:

**Test 1: Code Understanding** 

Prompt: "Summarize this project's architecture and main components in one sentence."

![](https://assets.zilliz.com/4_41e601fc82.png)

Qwen3-Coder-Plus nailed the summary—describing the project as a technical tutorial built on Milvus, with a focus on RAG systems, retrieval strategies, and more.

**Test 2: Code Generation**

Prompt: "Please create a small game of Tetris"

In under a minute, Qwen3-coder-plus:

- Autonomously installs required libraries

- Structures the game logic

- Creates a complete, playable implementation

- Handles all the complexity you'd normally spend hours researching

![](https://assets.zilliz.com/6_c67e1725eb.png)

![](https://assets.zilliz.com/7_fd91d5a290.gif)

This showcases true autonomous development—not just code completion, but architectural decision-making and complete solution delivery.


### Step 5: Set Up Your Vector Database

We’ll use [Zilliz Cloud](https://zilliz.com/cloud) as our vector database in this tutorial. 

**Create a Zilliz Cluster:**

1. Log into[ Zilliz Cloud Console](https://cloud.zilliz.com/)

2. Create a new cluster

3. Copy the **Public Endpoint** and **Token**

![](https://assets.zilliz.com/8_5e692e6e80.png)

![](https://assets.zilliz.com/9_753f281055.png)


### Step 6: Configure Code Context Integration

Create `~/.qwen/settings.json`:

```
{
  "mcpServers": {
    "code-context": {
      "command": "npx",
      "args": ["@zilliz/code-context-mcp@latest"],
      "env": {
        "OPENAI_API_KEY": "sk-xxxxxxxxxx",
        "MILVUS_ADDRESS": "https://in03-xxxx.cloud.zilliz.com",
        "MILVUS_TOKEN": "4f699xxxxx"
      },
      "cwd": "./server-directory",
      "timeout": 30000,
      "trust": false
    }
  }
}
```



### Step 7: Activate Enhanced Capabilities

Restart Qwen Code:

```
Qwen
```


Press **Ctrl + T** to see three new tools within our MCP server: 

- `index-codebase`: Creates semantic indexes for repository understanding

- `search-code`: Natural language code search across your codebase

- `clear-index`: Resets indexes when needed.

![](https://assets.zilliz.com/10_bebbb44460.png)


### Step 8: Test the Complete Integration

Here's a real example: In a big project, we reviewed code names and found that 'wider window' sounded unprofessional, so we decided to change it.

Prompt: "Find all functions related to 'wider window' that need professional renaming."

![](https://assets.zilliz.com/11_c54398c4f2.png)

As shown in the figure below, qwen3-coder-plus first called the `index_codebase` tool to create an index for the entire project.

![](https://assets.zilliz.com/12_25a7f3a039.png)

Then, the `index_codebase` tool created indexes for 539 files in this project, splitting them into 9,991 chunks. Immediately after building the index, it called the `search_code `tool to perform the query.

![](https://assets.zilliz.com/13_6766663346.png)

Next, it informed us that it found the corresponding files that needed modification.

![](https://assets.zilliz.com/14_7b3c7e9cc0.png)

Finally, it discovered 4 issues using Code Context, including functions, imports, and some naming in documentation, helping us complete this small task.

![](https://assets.zilliz.com/15_a529905b28.png)

With the addition of Code Context, `qwen3-coder-plus` now offers smarter code search and better understanding of coding environments. 


### What You've Built

You now have a complete AI coding copilot that combines:

- **Qwen3-Coder**: Intelligent code generation and autonomous development

- **Code Context**: Semantic understanding of existing codebases

- **Universal compatibility**: Works with Claude Code, Gemini CLI, VSCode, and more

This isn't just faster development—it enables entirely new approaches to legacy modernization, cross-team collaboration, and architectural evolution.


## Conclusion

As a developer, I’ve tried plenty of AI coding tools—from Claude Code to Cursor and Gemini CLI, and to Qwen Code—and while they’re great at generating new code, they usually fall flat when it comes to understanding existing codebases. That’s the real pain point: not writing functions from scratch, but navigating complex, messy, legacy code and figuring out _why_ things were done a certain way.

That’s what makes this setup with **Qwen3-Coder + Qwen Code+ Code Context** so compelling. You get the best of both worlds: a powerful coding model that can generate full-featured implementations _and_ a semantic search layer that actually understands your project history, structure, and naming conventions.

With vector search and the MCP plugin ecosystem, you're no longer stuck pasting random files into the prompt window or scrolling through your repo trying to find the right context. You just ask in plain language, and it finds the relevant files, functions, or decisions for you—like having a senior dev who remembers everything.

To be clear, this approach isn’t just faster—it actually changes how you work. It’s a step toward a new kind of development workflow where AI isn’t just a coding helper, but an architectural assistant, a teammate who gets the whole project context.

_That said... fair warning: Qwen3-Coder-Plus is amazing, but very token-hungry. Just building this prototype burned through 20 million tokens. So yeah—I’m now officially out of credits 😅_

__


Build a Coding Copilot with Qwen3-Coder & Code Context


Hands-on Tutorial: Build Your Own Coding Copilot with Qwen3-Coder, Qwen Code, and Code Context


Build a Production-Grade Chatbot with Kimi K2 and Milvus





## The AI Coding Boom—And Its Blind Spot

AI coding tools are everywhere—and they’re going viral for good reason. From [Claude Code, Gemini CLI](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md) to open-source Cursor alternatives, these agents can write functions, explain code dependency, and refactor entire files with a single prompt. Developers are racing to integrate them into their workflows, and in many ways, they’re delivering on the hype.

**But when it comes to _understanding your codebase_, most AI tools hit a wall.**

Ask Claude Code to find “where this project handles user authentication,” and it falls back on `grep -r "auth"`—spitting out 87 loosely related matches across comments, variable names, and filenames, likely missing many functions with authentication logic but not called “auth”. Try Gemini CLI, and it’ll look for keywords like “login” or “password,” missing functions like `verifyCredentials()` entirely. These tools are great at generating code, but when it’s time to navigate, debug, or explore unfamiliar systems, they fall apart. Unless they send the entire codebase to the LLM for context—burning through tokens and time—they struggle to provide meaningful answers.

_That’s the real gap in today’s AI tooling:_ **_code context._** 


## Cursor Nailed It—But Not for Everyone

**Cursor** tackles this head-on. Instead of keyword search, it builds a semantic map of your codebase using syntax trees, vector embeddings, and code-aware search. Ask it “where’s the email validation logic?” and it returns `isValidEmailFormat()` —not because the name matches, but because it understands what that code _does_.

While Cursor is powerful, it may not be suitable for everyone. **_Cursor is closed-source, cloud-hosted, and subscription-based._** That puts it out of reach for teams working with sensitive code, security-conscious organizations, indie developers, students, and anyone who prefers open systems.


## What if You Could Build Your Own Cursor?

Here’s the thing: the core technology behind Cursor isn’t proprietary. It’s built on proven open-source foundations—vector databases like [Milvus](https://milvus.io/), [embedding models](https://zilliz.com/ai-models), syntax parsers with Tree-sitter—all available to anyone willing to connect the dots.

_So, we asked:_ **_What if anyone could build their own Cursor?_** Runs on your infrastructure. No subscription fees. Fully customizable. Complete control over your code and data.

That’s why we built [**Code Context**](https://github.com/zilliztech/code-context)—an open-source, MCP-compatible plugin that brings powerful semantic code search to any AI coding agent, such as Claude Code and Gemini CLI, IDEs like VSCode, and even environments like Google Chrome. It also gives you the power to build your own coding agent like Cursor from scratch, unlocking real-time, intelligent navigation of your codebase.

**_No subscriptions. No black boxes. Just code intelligence—on your terms._**

In the rest of this post, we’ll walk through how Code Context works—and how you can start using it today.


## Code Context: Open-Source Alternative to Cursor's Intelligence

[**Code Context**](https://github.com/zilliztech/code-context) is an open-source, MCP-compatible semantic code search engine. Whether you're building a custom AI coding assistant from scratch or adding semantic awareness to AI coding agents like Claude Code and Gemini CLI, Code Context is the engine that makes it possible.

It runs locally, integrates with your favorite tools and environments, such as VS Code and Chrome browsers, and delivers robust code understanding without relying on cloud-only, closed-source platforms.

**Core capabilities include:**

- **Semantic Code Search via Natural Language:** Find code using plain English. Search for concepts like “user login verification” or “payment processing logic,” and Code Context locates the relevant functions—even if they don’t match the keywords exactly.

- **Multi-Language Support:** Search seamlessly across 15+ programming languages, including JavaScript, Python, Java, and Go, with consistent semantic understanding across them all.

- **AST-Based Code Chunking:** Code is automatically split into logical units, such as functions and classes, using AST parsing, ensuring search results are complete, meaningful, and never cut off mid-function.

- **Live, Incremental Indexing:** Code changes are indexed in real time. As you edit files, the search index stays up to date—no need for manual refreshes or re-indexing.

- **Fully Local, Secure Deployment:** Run everything on your own infrastructure. Code Context supports local models via Ollama and indexing via [Milvus](https://milvus.io/), so your code never leaves your environment.

- **First-Class IDE Integration:** The VSCode extension lets you search and jump to results instantly—right from your editor, with zero context switching.

- **MCP Protocol Support:** Code Context speaks MCP, making it easy to integrate with AI coding assistants and bring semantic search directly into their workflows.

- **Browser Plugin Support:** Search repositories directly from GitHub in your browser—no tabs, no copy-pasting, just instant context wherever you’re working.


### How Code Context Works

![](https://assets.zilliz.com/How_Code_Context_Works_3faaa2fff3.png)

Code Context uses a modular architecture with a core orchestrator and specialized components for embedding, parsing, storage, and retrieval. 


### The Core Module: Code Context Core 

At the heart of Code Context is the **Code Context Core**, which coordinates code parsing, embedding, storage, and semantic retrieval:

- **Text Processing Module** splits and parses code using Tree-sitter for language-aware AST analysis.

- **Embedding Interface** supports pluggable backends—currently OpenAI and VoyageAI—converting code chunks into vector embeddings that capture their semantic meaning and contextual relationships. 

- **The Vector Database Interface** stores these embeddings in a self-hosted [Milvus](https://milvus.io/) instance (by default) or in [Zilliz Cloud](https://zilliz.com/cloud), the managed version of Milvus. 

All of this is synchronized with your file system on a scheduled basis, ensuring the index stays up to date without requiring manual intervention.


### Extension Modules on top of Code Context Core

- **VSCode Extension**: Seamless IDE integration for fast in-editor semantic search and jump-to-definition.

- **Chrome Extension**: Inline semantic code search while browsing GitHub repositories—no need to switch tabs.

- **MCP Server**: Exposes Code Context to any AI coding assistants via the MCP protocol, enabling real-time, context-aware assistance.


## Getting Started with Code Context 

Code Context can be plugged into coding tools you already use or to build a custom AI coding assistant from scratch. In this section, we’ll walk through both scenarios:

- How to integrate Code Context with existing tools 

- How to set up the Core Module for standalone semantic code search when building your own AI coding assistant 


### MCP Integration

Code Context supports **Model Context Protocol (MCP)**, allowing AI coding agents like Claude Code to use it as a semantic backend.

To integrate with Claude Code:

```
claude mcp add code-context -e OPENAI_API_KEY=your-openai-api-key -e MILVUS_ADDRESS=your-zilliz-cloud-public-endpoint -e MILVUS_TOKEN=your-zilliz-cloud-api-key -- npx @zilliz/code-context-mcp@latest
```

![](https://assets.zilliz.com/mcp_code_context_processed_3908097481.gif)

Once configured, Claude Code will automatically call Code Context for semantic code search when needed.

To integrate with other tools or environments, check out our[ GitHub repo](https://github.com/zilliztech/code-context) for more examples and adapters.

![](https://assets.zilliz.com/MCP_Integration_2_683c7ef73d.png)


### Building Your Own AI Coding Assistant with Code Context

To build a custom AI assistant using Code Context, you'll set up the core module for semantic code search in just three steps:

1. Configure your embedding model

2. Connect to your vector database

3. Index your project and start searching

Here’s an example using **OpenAI Embeddings** and [**Zilliz Cloud**](https://zilliz.com/cloud) **vector database** as the vector backend:

```
import { CodeContext, MilvusVectorDatabase, OpenAIEmbedding } from '@zilliz/code-context-core';

// Initialize embedding model
const embedding = new OpenAIEmbedding({
    apiKey: 'your-openai-api-key',
    model: 'text-embedding-3-small'
});

// Initialize Zilliz Cloud vector database
// Sign up for free at https://zilliz.com/cloud
const vectorDatabase = new MilvusVectorDatabase({
    address: 'https://xxx-xxxxxxxxxxxx.serverless.gcp-us-west1.cloud.zilliz.com',
    token: 'xxxxxxx'
});

// Create the Code Context indexer
const context = new CodeContext({ embedding, vectorDatabase });

// Index the codebase
await context.indexCodebase('./your-project');

// Perform semantic code search
const results = await context.semanticSearch('./your-project', 'vector database operations', 5);
results.forEach(result => {
    console.log(`${result.relativePath}:${result.startLine}-${result.endLine}`);
    console.log(`score: ${(result.score * 100).toFixed(2)}%`);
});
```



### VSCode Extension

Code Context is available as a VSCode extension named **"Semantic Code Search"**, which brings intelligent, natural language–driven code search directly into your editor.

![](https://assets.zilliz.com/VS_Code_Extension_e358f36464.png)


Once installed:

- Configure your API key

- Index your project

- Use plain English queries (no exact match needed)

- Jump to results instantly with click-to-navigate

![](https://assets.zilliz.com/mcp_code_context_processed_3908097481.gif)


This makes semantic exploration a native part of your coding workflow—no terminal or browser required.


### Chrome Extension (Coming Soon)

Our upcoming **Chrome extension** brings Code Context to GitHub web pages, allowing you to run semantic code search directly inside any public repository—no context switching or tabs required.

![](https://assets.zilliz.com/chrome_4e67b683d7.gif)

You’ll be able to explore unfamiliar codebases with the same deep search capabilities you have locally. Stay tuned—the extension is in development and launching soon.


## Why Use Code Context?

The basic setup gets you running quickly, but where **Code Context** truly shines is in professional, high-performance development environments. Its advanced features are designed to support serious workflows, from enterprise-scale deployments to custom AI tooling.


### Private Deployment for Enterprise-Grade Security

Code Context supports fully offline deployment using the **Ollama** local embedding model and **Milvus** as a self-hosted vector database. This enables an entirely private code search pipeline: no API calls, no internet transmission, and no data ever leaves your local environment.

This architecture is ideal for industries with strict compliance requirements—such as finance, government, and defense—where code confidentiality is non-negotiable.


### Real-Time Indexing with Intelligent File Sync

Keeping your code index up to date shouldn’t be slow or manual. Code Context includes a **Merkle Tree–based file monitoring system** that detects changes instantly and performs incremental updates in real time.

![](https://assets.zilliz.com/workflow_0fd958fe81.jpeg)

By only re-indexing modified files, it reduces update times for large repositories from minutes to seconds. This ensures the code you just wrote is already searchable, without needing to click “refresh.”

In fast-paced dev environments, that kind of immediacy is critical.


### AST Parsing That Understands Code Like You Do

Traditional code search tools split text by line or character count, often breaking logical units and returning confusing results.

![](https://assets.zilliz.com/AST_Parsing_That_Understands_Code_Like_You_Do_3236afc075.png)

Code Context does better. It utilizes Tree-sitter AST parsing to comprehend the actual code structure. It identifies complete functions, classes, interfaces, and modules, providing clean, semantically complete results.

It supports major programming languages, including JavaScript/TypeScript, Python, Java, C/C++, Go, and Rust, with language-specific strategies for accurate chunking. For unsupported languages, it falls back to rule-based parsing, ensuring graceful handling without crashes or empty results.

These structured code units also feed into metadata for more accurate semantic search.


### Open Source and Extensible by Design

Code Context is fully open source under the MIT license. All core modules are publicly available on GitHub.

We believe open infrastructure is the key to building powerful, trustworthy developer tools—and invite developers to extend it for new models, languages, or use cases.


### Solving the Context Window Problem for AI Assistants

Large language models (LLMs) have a hard limit: their context window. This restricts them from seeing an entire codebase, which reduces the accuracy of completions, fixes, and suggestions.

Code Context helps bridge that gap. Its semantic code search retrieves the _right_ pieces of code, giving your AI assistant focused, relevant context to reason with. It improves the quality of AI-generated output by letting the model “zoom in” on what actually matters.

Popular AI coding tools, such as Claude Code and Gemini CLI, lack native semantic code search—they rely on shallow, keyword-based heuristics. Code Context, when integrated via **MCP**, gives them a brain upgrade.


### Built for Developers, by Developers

Code Context is packaged for modular reuse: each component is available as an independent **npm** package. You can mix, match, and extend as needed for your project.

- Need only a semantic code search? Use`@zilliz/code-context-core`

- Want to plug into an AI agent? Add `@zilliz/code-context-mcp`

- Building your own IDE/browser tool? Fork our VSCode and Chrome extension examples

Some example applications of code context:

- **Context-aware autocomplete plugins** that pull relevant snippets for better LLM completions

- **Intelligent bug detectors** that gather surrounding code to improve fix suggestions

- **Safe code refactoring tools** that find semantically related locations automatically

- **Architecture visualizers** that build diagrams from semantic code relationships

- **Smarter code review assistants** that surface historical implementations during PR reviews


## Welcome to Join Our Community

[**Code Context**](https://github.com/zilliztech/code-context) is more than just a tool—it’s a platform for exploring how **AI and vector databases** can work together to truly understand code. As AI-assisted development becomes the norm, we believe semantic code search will be a foundational capability.

We welcome contributions of all kinds:

- Support for new languages

- New embedding model backends

- Innovative AI-assisted workflows

- Feedback, bug reports, and design ideas

Find us here:

- [Code Context on GitHub](https://github.com/zilliztech/code-context) | [**MCP npm package**](https://www.npmjs.com/package/@zilliz/code-context-mcp) | [**VSCode marketplace**](https://marketplace.visualstudio.com/items?itemName=zilliz.semanticcodesearch)

- [Discord](https://discuss.milvus.io/) | [LinkedIn](https://www.linkedin.com/company/the-milvus-project/) | [X](https://x.com/zilliz_universe) | [YouTube](https://www.youtube.com/@MilvusVectorDatabase/featured)

Together, we can build the infrastructure for the next generation of AI development tools—transparent, powerful, and developer-first.

[![](https://assets.zilliz.com/office_hour_83d4623510.png)](https://meetings.hubspot.com/chloe-williams1/milvus-office-hour)


Building an Open-Source Alternative to Cursor with Code Context



Every day on tech “X” (formerly Twitter), you see developers showing off their setups—automated deployment pipelines that handle complex multi-environment releases without a hitch; monitoring systems that intelligently route alerts to the right team members based on service ownership; development workflows that automatically sync GitHub issues with project management tools and notify stakeholders at exactly the right moments.

These seemingly "advanced" operations all share the same secret: **workflow automation tools.**

Think about it. A pull request gets merged, and the system automatically triggers tests, deploys to staging, updates the corresponding Jira ticket, and notifies the product team in Slack. A monitoring alert fires, and instead of spamming everyone, it intelligently routes to the service owner, escalates based on severity, and automatically creates incident documentation. A new team member joins, and their development environment, permissions, and onboarding tasks get provisioned automatically.

These integrations that used to require custom scripts and constant maintenance now run themselves 24/7 once you set them up properly. 

Recently, I discovered [N8N](https://github.com/Zie619/n8n-workflows), a visual workflow automation tool, and more importantly, stumbled upon an open-source repository containing over 2000 ready-to-use workflow templates. This post will walk you through what I learned about workflow automation, why N8N caught my attention, and how you can leverage these pre-built templates to set up sophisticated automation in minutes instead of building everything from scratch.


## Workflow: Let Machines Handle the Grunt Work

### What is workflow?

At its core, workflow is just a set of automated task sequences. Picture this: you take a complex process and break it down into smaller, manageable chunks. Each chunk becomes a "node" that handles one specific job—maybe calling an API, processing some data, or sending a notification. String these nodes together with some logic, add a trigger, and you've got a workflow that runs itself.

Here's where it gets practical. You can set up workflows to automatically save email attachments to Google Drive when they arrive, scrape website data on a schedule and dump it into your database, or route customer tickets to the right team members based on keywords or priority levels.


### Workflow vs AI Agent: Different Tools for Different Jobs

Before we go further, let's clear up some confusion. A lot of developers mix up workflows with AI agents, and while both can automate tasks, they're solving completely different problems.

- **Workflows** follow predefined steps with no surprises. They're triggered by specific events or schedules and are perfect for repetitive tasks with clear steps like data syncing and automated notifications.

- **AI Agents** make decisions on the fly and adapt to situations. They continuously monitor and decide when to act, making them ideal for complex scenarios requiring judgment calls like chatbots or automated trading systems.

| **What We're Comparing** | **Workflows**                          | **AI Agents**                                    |
| ------------------------ | -------------------------------------- | ------------------------------------------------ |
| How It Thinks            | Follows predefined steps, no surprises | Makes decisions on the fly, adapts to situations |
| What Triggers It         | Specific events or schedules           | Continuously monitors and decides when to act    |
| Best Used For            | Repetitive tasks with clear steps      | Complex scenarios requiring judgment calls       |
| Real-World Examples      | Data syncing, automated notifications  | Chatbots, automated trading systems              |

For most of the automation headaches you face daily, workflows will handle about 80% of your needs without the complexity.


## Why N8N Caught My Attention

The workflow tool market is pretty crowded, so why did N8N catch my attention? It all comes down to one key advantage: [**N8N**](https://github.com/Zie619/n8n-workflows) **uses a graph-based architecture that actually makes sense for how developers think about complex automation.**


### Why Visual Representation Actually Matters for Workflows 

N8N lets you build workflows by connecting nodes on a visual canvas. Each node represents a step in your process, and the lines between them show how data flows through your system. This isn't just eye candy—it's a fundamentally better way to handle complex, branching automation logic.

![](https://assets.zilliz.com/n8n1_3bcae91c82.png)

N8N brings enterprise-grade capabilities to the table with integrations for over 400 services, complete local deployment options for when you need to keep data in-house, and robust error handling with real-time monitoring that actually helps you debug issues instead of just telling you something broke.

![](https://assets.zilliz.com/n8n2_248855922d.png)

### N8N Has 2000+ Ready-Made Templates

The biggest barrier to adopting new tools isn't learning the syntax—it's figuring out where to start. Here's where I discovered this open-source project '[n8n-workflows](https://github.com/Zie619/n8n-workflows)' that became invaluable. It contains 2,053 ready-to-use workflow templates that you can deploy and customize immediately.


## Getting Started with N8N

Now let’s walk through how to use N8N. It is pretty easy. 


### Environment Setup

I assume most of you have a basic environment setup. If not, check the official resources:

- Docker website: https://www.docker.com/

- Milvus website: https://milvus.io/docs/prerequisite-docker.md

- N8N website: https://n8n.io/

- Python3 website: https://www.python.org/

- N8n-workflows: https://github.com/Zie619/n8n-workflows


### Clone and Run the Template Browser

```
git clone https://github.com/Zie619/n8n-workflows.git
pip install -r requirements.txt
python run.py
http://localhost:8000
```

![](https://assets.zilliz.com/n8n3_0db8e22872.png)


![](https://assets.zilliz.com/n8n4_b6b9ba6635.png)

### Deploy N8N

```
docker run -d -it --rm --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n -e N8N_SECURE_COOKIE=false -e N8N_HOST=192.168.4.48 -e N8N_LISTEN_ADDRESS=0.0.0.0  n8nio/n8n:latest
```

**⚠️ Important:** Replace N8N_HOST with your actual IP address

![](https://assets.zilliz.com/n8n5_6384caa548.png)

### Importing Templates 

Once you find a template you want to try, getting it into your N8N instance is straightforward:

![](https://assets.zilliz.com/n8n6_2ea8b14bd9.png)

#### **1. Download the JSON File**

Each template is stored as a JSON file that contains the complete workflow definition.

![](https://assets.zilliz.com/n8n7_d58242d81a.png)

#### **2. Open N8N Editor**

Navigate to Menu → Import Workflow

![](https://assets.zilliz.com/n8n8_9961929091.png)

#### **3. Import the JSON**

Select your downloaded file and click Import

![](https://assets.zilliz.com/n8n9_3882b6ade6.png)

From there, you just need to adjust the parameters to match your specific use case. You'll have a professional-grade automation system running in minutes instead of hours.

With your basic workflow system up and running, you might be wondering how to handle more complex scenarios that involve understanding content rather than just processing structured data. That's where vector databases come into play. 


## Vector Databases: Making Workflows Smart with Memory

Modern workflows need to do more than just shuffle data around. You're dealing with unstructured content—documentation, chat logs, knowledge bases—and you need your automation to actually understand what it's working with, not just match exact keywords.


### Why Your Workflow Needs Vector Search

Traditional workflows are basically pattern matching on steroids. They can find exact matches, but they can't understand context or meaning.

When someone asks a question, you want to surface all the relevant information, not just documents that happen to contain the exact words they used.

This is where[ vector databases](https://zilliz.com/learn/what-is-vector-database) like [**Milvus**](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/cloud) come in. Milvus gives your workflows the ability to understand semantic similarity, which means they can find related content even when the wording is completely different. 

Here's what Milvus brings to your workflow setup: 

- **Massive scale storage** that can handle billions of vectors for enterprise knowledge bases

- **Millisecond-level search performance** that won't slow down your automation

- **Elastic scaling** that grows with your data without requiring a complete rebuild

The combination transforms your workflows from simple data processing into intelligent knowledge services that can actually solve real problems in information management and retrieval.


## What This Actually Means for Your Development Work

Workflow automation isn't rocket science—it's about making complex processes simple and repetitive tasks automatic. The value is in the time you get back and the errors you avoid.

Compared to enterprise solutions that cost tens of thousands of dollars, open-source N8N offers a practical path forward. The open-source version is free, and the drag-and-drop interface means you don't need to write code to build sophisticated automation.

Together with Milvus for intelligent search capabilities, workflow automation tools like N8N upgrade your workflows from simple data processing to smart knowledge services that solve real problems in information management and retrieval.

The next time you find yourself doing the same task for the third time this week, remember: there's probably a template for that. Start small, automate one process, and watch as your productivity multiplies while your frustration disappears.


Boost Efficiency with N8N Workflow Automation


I Discovered This N8N Repo That Actually 10x'd My Workflow Automation Efficiency





Your IDE is bloated. Your coding assistant is outdated. And you're still stuck right-clicking to refactor? Welcome to the CLI renaissance.

AI code assistants are evolving from gimmicks to go-to tools and developers are taking sides. Beyond the startup sensation Cursor, **Anthropic’s** [**Claude Code**](https://www.anthropic.com/claude-code) brings precision and polish. Google’s [**Gemini CLI**](https://github.com/google-gemini/gemini-cli)? Fast, free, and hungry for context. Both promise to make natural language the new shell scripting. So which one should _you_ trust to refactor your next repo?

From what I’ve seen, Claude Code had the early lead. But the game changed fast. After Gemini CLI launched, developers flocked to it—**racking up 15.1k GitHub stars within 24 hours.** As of now, it’s soared past **55,000 stars** and counting. Amazing!

Here's my quick takeaway on why so many developers are excited about Gemini CLI:

- **It's open source under Apache 2.0 and completely free:** Gemini CLI connects to Google's top-tier Gemini 2.0 Flash model at no cost. Simply log in with your personal Google account to access Gemini Code Assist. During the preview period, you get up to 60 requests per minute and 1,000 daily requests—all free of charge.

- **It's a true multi-task powerhouse:** Beyond programming (its strongest suit), it handles file management, content generation, script control, and even Deep Research capabilities.

- **It's lightweight:** You can seamlessly embed it in terminal scripts or use it as a standalone agent.

- **It offers long context length:** With 1 million tokens of context (roughly 750,000 words), it can ingest entire codebases for smaller projects in a single pass.


## Why Developers Are Ditching IDEs for AI-Powered Terminals

Why is there such enthusiasm for these terminal-based tools? As developers, you've probably felt this pain: Traditional IDEs pack impressive features, but they come with workflow complexity that kills momentum. Want to refactor a single function? You need to select the code, right-click for the context menu, navigate to "Refactor," choose the specific refactoring type, configure options in a dialog box, and finally apply changes.

**Terminal AI tools have changed this workflow by streamlining all operations into natural language commands.** Instead of memorizing command syntax, you simply say: "_Help me refactor this function to improve readability,_" and watch as the tool handles the entire process.

This isn't just convenience—it's a fundamental shift in how we think. Complex technical operations become natural language conversations, freeing us to focus on business logic rather than the mechanics of tools.


## Claude Code or Gemini CLI? Choose Your Co-Pilot Wisely

Since Claude Code is also quite popular and easy to use and has previously dominated adoption, how does it compare to the new Gemini CLI? How should we choose between the two? Let's take a closer look at these AI coding tools.


### **1. Cost: Free vs Paid**

- **Gemini CLI** is completely free with any Google account, providing 1,000 requests per day and 60 requests per minute, with no billing setup required.

- **Claude Code** requires an active Anthropic subscription and follows a pay-per-use model, but includes enterprise-level security and support that's valuable for commercial projects.


### **2. Context Window: How Much Code Can It See?**

- **Gemini CLI:** 1 million tokens (roughly 750,000 words)

- **Claude Code:** Approximately 200,000 tokens (about 150,000 words)

Larger context windows enable models to reference more input content when generating responses. They also help maintain conversation coherence in multi-turn dialogues, giving the model better memory of your entire conversation.

Essentially, Gemini CLI can analyze your entire small-to-medium project in a single session, making it ideal for comprehending large codebases and cross-file relationships. Claude Code works better when you're focusing on specific files or functions.


### **3. Code Quality vs Speed**

|                           |                |                 |                                      |
| ------------------------- | -------------- | --------------- | ------------------------------------ |
| **Feature**               | **Gemini CLI** | **Claude Code** | **Notes**                            |
| **Coding speed**          | 8.5/10         | 7.2/10          | Gemini generates code faster         |
| **Coding quality**        | 7.8/10         | 9.1/10          | Claude generates higher quality code |
| **Error handling**        | 7.5/10         | 8.8/10          | Claude is better at error handling   |
| **Context understanding** | 9.2/10         | 7.9/10          | Gemini has longer memory             |
| **Multilingual support**  | 8.9/10         | 8.5/10          | Both are excellent                   |

- **Gemini CLI** generates code faster and excels at understanding large contexts, making it great for rapid prototyping.

- **Claude Code** nails precision and error handling, making it better suited for production environments where code quality is critical.


### **4. Platform Support: Where Can You Run It?**

- **Gemini CLI** works equally well across Windows, macOS, and Linux from day one.

- **Claude Code** was optimized for macOS first, and while it runs on other platforms, the best experience is still on Mac.


### **5. Authentication and Access**

**Claude Code** requires an active Anthropic subscription (Pro, Max, Team, or Enterprise) or API access through AWS Bedrock/Vertex AI. This means you need to set up billing before you can start using it.

**Gemini CLI** offers a generous free plan for individual Google account holders, including 1,000 free requests per day and 60 requests per minute to the full-featured Gemini 2.0 Flash model. Users requiring higher limits or specific models can upgrade via API keys.


### **6. Feature Comparison Overview**

|                       |                  |                               |
| --------------------- | ---------------- | ----------------------------- |
| **Feature**           | **Claude Code**  | **Gemini CLI**                |
| Context Window Length | 200K tokens      | 1M tokens                     |
| Multimodal Support    | Limited          | Powerful (images, PDFs, etc.) |
| Code Understanding    | Excellent        | Excellent                     |
| Tool Integration      | Basic            | Rich (MCP Servers)            |
| Security              | Enterprise-grade | Standard                      |
| Free Requests         | Limited          | 60/min, 1000/day              |


## When to Choose Claude Code vs Gemini CLI?

Now that we've compared the key features of both tools, here are my takeaways on when to choose each:

**Choose Gemini CLI if:**

- Cost-effectiveness and rapid experimentation are priorities

- You're working on large projects that need massive context windows

- You love cutting-edge, open-source tools

- Cross-platform compatibility is crucial

- You want powerful multimodal capabilities

**Choose Claude Code if:**

- You need high-quality code generation

- You're building mission-critical commercial applications

- Enterprise-level support is non-negotiable

- Code quality trumps cost considerations

- You're primarily working on macOS


## Claude Code vs. Gemini CLI: Setup and Best Practices

Now that we have a basic understanding of the capabilities of these two terminal AI tools, let's take a closer look at how to get started with them and the best practices.


### Claude Code Setup and Best Practices

**Installation:** Claude Code requires npm and Node.js version 18 or higher.

```
# Install Claude Code on your system
npm install -g @anthropic-ai/claude-code

# Set up API key
claude config set api-key YOUR_API_KEY

# Verify installation was successful
claude --version

# Launch Claude Code
Claude
```


****![](https://assets.zilliz.com/1_c413bbf950.png)****

**Best Practices for Claude Code:**

1. **Start with architecture understanding:** When approaching a new project, have Claude Code help you understand the overall structure first using natural language.

```
# Let Claude analyze project architecture
> Analyze the main architectural components of this project

# Understand security mechanisms
> What security measures does this system have?

# Get code overview
> Give me an overview of this codebase
```

2. **Be specific and provide context:** The more context you give, the more accurate Claude Code's suggestions will be.

```
# Implement specific features
> Implement an initial version for GitHub issue #123

# Code migration
> Help me migrate this codebase to the latest Java version, first create a plan

# Code refactoring
> Refactor this function to make it more readable and maintainable
```

3. **Use it for debugging and optimization:**

```
# Error analysis
> What caused this error? How can we fix it?

# Performance optimization
> Analyze the performance bottlenecks in this code

# Code review
> Review this pull request and point out potential issues
```

**Summary:**

- Use progressive learning by starting with simple code explanations, then gradually moving to more complex code generation tasks

- Maintain conversation context since Claude Code remembers earlier discussions

- Provide feedback using the `bug` command to report issues and help improve the tool

- Stay security-conscious by reviewing data collection policies and exercising caution with sensitive code


### Gemini CLI Setup and Best Practices

**Installation:** Like Claude Code, Gemini CLI requires npm and Node.js version 18 or higher.

```
# Install Gemini CLI
npm install -g @google/gemini-cli

# Login to your Google account
gemini auth login

# Verify installation
gemini --version

# Launch Gemini CLI
Gemini
```


![](https://assets.zilliz.com/2_bec1984bb0.png)

If you have a personal account, log in with your Google account for immediate access, with a limit of 60 requests per minute. For higher limits, configure your API key:

```
export GEMINI_API_KEY="YOUR_API_KEY"
```


**Best Practices for Gemini CLI:**

1. **Start with architecture understanding:** Like Claude Code, when approaching a new project, have Gemini CLI help you understand the overall structure first using natural language. Note that Gemini CLI supports a 1 million token context window, making it highly effective for large-scale codebase analysis.

```
# Analyze project architecture
> Analyze the main architectural components of this project

# Understand security mechanisms
> What security measures does this system have?

# Get code overview
> Give me an overview of this codebase
```

2. **Leverage its multimodal capabilities:** This is where Gemini CLI truly shines.

```
# Generate app from PDF
> Create a new app based on this PDF design document

# Generate code from sketch
> Generate frontend code based on this UI sketch

# Image processing tasks
> Convert all images in this directory to PNG format and rename using EXIF data
```

3. **Explore tool integrations:** Gemini CLI can integrate with multiple tools and MCP servers for enhanced functionality.

```
# Connect external tools
> Use MCP server to connect my local system tools

# Media generation
> Use Imagen to generate project logo

# Search integration
> Use Google search tool to find related technical documentation
```

**Summary:**

- Be project-oriented: Always launch Gemini from your project directory for better contextual understanding

- Maximize multimodal features by using images, documents, and other media as inputs, not just text

- Explore tool integrations by connecting external tools with MCP servers

- Enhance search capabilities by using built-in Google search for up-to-date information


## AI Code is Outdated on Arrival. Here’s How to Fix it with Milvus

_AI coding tools like Claude Code and Gemini CLI are powerful—but they have a blind spot:_ **_they don’t know what’s current_**_._

_The reality? Most models generate outdated patterns straight out of the box. They were trained months ago, sometimes years. So while they can generate code quickly, they can’t guarantee that it reflects_ **_your latest APIs_**_, frameworks, or SDK versions._

**Real example:**

Ask Cursor how to connect to Milvus, and you might get this:

```
connections.connect("default", host="localhost", port="19530")
```

Looks fine, except that method’s now deprecated. The recommended approach is to use  `MilvusClient` but most assistants don’t know that yet. 

Or take OpenAI’s own API. Many tools still suggest `gpt-3.5-turbo` via `openai.ChatCompletion`, a method deprecated in March 2024. It’s slower, costs more, and delivers worse results. But the LLM doesn’t know that. 

![](https://assets.zilliz.com/3_8f0d1a42b6.png)

![](https://assets.zilliz.com/3_5_3f4c4a0d4c.png)


### The Fix: Real-Time Intelligence with Milvus MCP + RAG

To solve this, we combined two powerful ideas:

- **Model Context Protocol (MCP)**: A standard for agentic tools to interact with live systems through natural language

- **Retrieval-Augmented Generation (RAG)**: Fetches the freshest, most relevant content—on demand

Together, they make your assistant smarter and current.

![](https://assets.zilliz.com/4_e6bc6cacd6.png)

**Here’s how it works:**

1. Preprocess your documentation, SDK references, and API guides

2. Store them as vector embeddings in [**Milvus**](https://milvus.io/), our open-source vector database

3. When a dev asks a question (e.g. “How do I connect to Milvus?”), the system:

   - Runs a **semantic search**

   - Retrieves the most relevant docs and examples
   
   - Injects them into the assistant’s prompt context



4. Result: code suggestions that reflect **exactly what’s true right now**


### Live Code, Live Docs

With the **Milvus MCP Server**, you can plug this flow directly into your coding environment. Assistants get smarter. Code gets better. Devs stay in flow.

And it’s not just theoretical—we’ve battle-tested this against other setups like Cursor’s Agent Mode, Context7, and DeepWiki. The difference? Milvus + MCP doesn't just summarize your project—it stays in sync with it.

See it in Action: [Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP ](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)


## The Future of Coding is Conversational—And It's Happening Right Now

The terminal AI revolution is just beginning. As these tools mature, we'll likely see even tighter integration with development workflows, better code quality, and solutions to the currency problem through approaches like MCP+RAG.

Whether you choose Claude Code for its quality or Gemini CLI for its accessibility and power, one thing is clear: **natural language programming is here to stay.** The question isn't whether to adopt these tools, but how to integrate them effectively into your development workflow.

We're witnessing a fundamental shift from memorizing syntax to having conversations with our code. **The future of coding is conversational—and it's happening right now in your terminal.**


## Keep Reading

- [Building a Production-Ready AI Assistant with Spring Boot and Milvus](https://milvus.io/blog/building-a-production-ready-ai-assistant-with-spring-boot-and-milvus.md)

- [Zilliz MCP Server: Natural Language Access to Vector Databases - Zilliz blog](https://zilliz.com/blog/introducing-zilliz-mcp-server)

- [VDBBench 1.0: Real-World Benchmarking for Vector Databases - Milvus Blog](https://milvus.io/blog/vdbbench-1-0-benchmarking-with-your-real-world-production-workloads.md)

- [Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)

- [Why AI Databases Don't Need SQL ](https://milvus.io/blog/why-ai-databases-do-not-need-sql.md)


Claude Code vs Gemini CLI: Who’s the Real Dev Co-Pilot?


Claude Code vs Gemini CLI: Which One’s the Real Dev Co-Pilot?





Most vector database benchmarks test with static data and pre-built indexes. But production systems don't work that way—data flows continuously while users run queries, filters fragment indexes, and performance characteristics shift dramatically under concurrent read/write loads.

Today we're releasing [**VDBBench 1.0**](https://github.com/zilliztech/VectorDBBench/releases/tag/v1.0.0), an open-source benchmark designed from the ground up to test vector databases under realistic production conditions: streaming data ingestion, metadata filtering with varying selectivity, and concurrent workloads that reveal actual system bottlenecks.

[**Download VDBBench 1.0 →**](https://github.com/zilliztech/VectorDBBench/releases/tag/v1.0.0) |[ **View Leaderboard →**](https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch)


## Why Current Benchmarks Are Misleading 

Let's be honest—there's a strange phenomenon in our industry. Everyone talks about "not gaming benchmarks," yet many participate in exactly that behavior. Since the vector database market exploded in 2023, we've seen numerous examples of systems that "benchmark beautifully" but "fail miserably" in production, wasting engineering time and damaging project credibility.

We've witnessed this disconnect firsthand. For example, Elasticsearch boasts millisecond-level query speeds, but behind the scenes, it can take over 20 hours just to optimize its index. What production system can tolerate such downtime?

The problem stems from three fundamental flaws:

- **Outdated datasets:** Many benchmarks still rely on legacy datasets like SIFT (128 dimensions) while modern embeddings range from 768-3,072 dimensions. The performance characteristics of systems operating on 128D vs. 1024D+ vectors are fundamentally different—memory access patterns, index efficiency, and computational complexity all change dramatically.

- **Vanity metrics:** Benchmarks focus on average latency or peak QPS, creating a distorted picture. A system with 10ms average latency but 2-second P99 latency creates a terrible user experience. Peak throughput measured over 30 seconds tells you nothing about sustained performance.

- **Oversimplified scenarios:** Most benchmarks test basic "write data, build index, query" workflows—essentially "Hello World" level testing. Real production involves continuous data ingestion while serving queries, complex metadata filtering that fragments indexes, and concurrent read/write operations competing for resources.


## What’s New in VDBBench 1.0?

VDBBench doesn't just iterate on outdated benchmarking philosophies—it rebuilds the concept from first principles with one guiding belief: a benchmark is only valuable if it predicts actual production behavior.

We've engineered VDBBench to faithfully replicate real-world conditions across three critical dimensions: **data authenticity, workload patterns, and performance measurement methodologies.**

Let’s take a closer look at what new features are brought to the table. 


### **🚀 Redesigned Dashboard with Production-Relevant Visualizations**

Most benchmarks focus only on raw data output, but what matters is how engineers interpret and act on those results. We redesigned the UI to prioritize clarity and interactivity—enabling you to spot performance gaps between systems and make fast infrastructure decisions.

The new dashboard visualizes not just performance numbers, but the relationships between them: how QPS degrades under different filter selectivity levels, how recall fluctuates during streaming ingestion, and how latency distributions reveal system stability characteristics.
![](https://assets.zilliz.com/vdb_1_df593dea0b.png)

We've retested major vector database platforms including **Milvus, Zilliz Cloud, Elastic Cloud, Qdrant Cloud, Pinecone, and OpenSearch** with their latest configurations and recommended settings, ensuring all benchmark data reflects current capabilities. All testing results are available at the[ VDBBench Leaderboard](https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch).


### 🏷️ Tag Filtering: The Hidden Performance Killer

Real-world queries rarely happen in isolation. Applications combine vector similarity with metadata filtering ("find shoes that look like this photo but cost under $100"). This filtered vector search creates unique challenges that most benchmarks completely ignore.

Filtered searches introduce complexity in two critical areas:

- **Filter Complexity**: More scalar fields and complex logical conditions increase computational demands and can cause insufficient recall and graph index fragmentation.

- **Filter Selectivity**: This is the "hidden performance killer" we've repeatedly verified in production. When filtering conditions become highly selective (filtering out 99%+ of data), query speeds can fluctuate by orders of magnitude, and recall can become unstable as index structures struggle with sparse result sets.

VDBBench systematically tests various filtering selectivity levels (from 50% to 99.9%), providing a comprehensive performance profile under this critical production pattern. The results often reveal dramatic performance cliffs that would never show up in traditional benchmarks.

**Example**: In Cohere 1M tests, Milvus maintained consistently high recall across all filter selectivity levels, while OpenSearch exhibited unstable performance with recall fluctuating significantly under different filtering conditions—falling below 0.8 recall in many cases, which is unacceptable for most production environments.

![](https://assets.zilliz.com/vdb_2_0ef89463e5.png)

_Figure: QPS and Recall of Milvus and OpenSearch Across Different Filter Selectivity Levels (Cohere 1M Test)._ 


### 🌊 Streaming Read/Write: Beyond Static Index Testing

Production systems rarely enjoy the luxury of static data. New information continuously flows in while searches execute—a scenario where many otherwise impressive databases collapse under the dual pressure of maintaining search performance while handling continuous writes.

VDBBench's streaming scenarios simulate real parallel operations, helping developers understand system stability in high-concurrency environments, particularly how data writing impacts query performance and how performance evolves as data volume increases.

To ensure fair comparisons across different systems, VDBBench uses a structured approach:

- Configure controlled write rates that mirror target production workloads (e.g., 500 rows/sec distributed across 5 parallel processes)

- Trigger search operations after every 10% of data ingestion, alternating between serial and concurrent modes

- Record comprehensive metrics: latency distributions (including P99), sustained QPS, and recall accuracy

- Track performance evolution over time as data volume and system stress increase

This controlled, incremental load testing reveals how well systems maintain stability and accuracy under ongoing ingestion—something traditional benchmarks rarely capture.

**Example**: In Cohere 10M streaming tests, Pinecone maintained higher QPS and recall throughout the write cycle compared to Elasticsearch. Notably, Pinecone's performance significantly improved after ingestion completion, demonstrating strong stability under sustained load, while Elasticsearch showed more erratic behavior during active ingestion phases.
![](https://assets.zilliz.com/vdb3_9d2a5298b0.png)

Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test (500 rows/s Ingestion Rate). 

VDBBench goes even further by supporting an optional optimization step, allowing users to compare streaming search performance before and after index optimization. It also tracks and reports the actual time spent on each stage, offering deeper insights into system efficiency and behavior under production-like conditions.
![](https://assets.zilliz.com/vdb4_0caee3b201.png)

_Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test After Optimization (500 rows/s Ingestion Rate)_

As shown in our tests, Elasticsearch surpassed Pinecone in QPS—after index optimization. But when the x-axis reflects actual elapsed time, it's clear that Elasticsearch took significantly longer to reach that performance. In production, that delay matters. This comparison reveals a key tradeoff: peak throughput vs. time-to-serve.


### 🔬 Modern Datasets That Reflect Current AI Workloads

We've completely overhauled the datasets used for vector database benchmarking. Instead of legacy test sets like SIFT and GloVe, VDBBench uses vectors generated from state-of-the-art embedding models like OpenAI and Cohere that power today's AI applications.

To ensure relevance, especially for use cases like Retrieval-Augmented Generation (RAG), we selected corpora that reflect real-world enterprise and domain-specific scenarios:

|            |                     |                |                 |                              |
| ---------- | ------------------- | -------------- | --------------- | ---------------------------- |
| **Corpus** | **Embedding Model** | **Dimensions** | **Size**        | **Use Case**                 |
| Wikipedia  | Cohere V2           | 768            | 1M / 10M        | General knowledge base       |
| BioASQ     | Cohere V3           | 1024           | 1M / 10M        | Domain-specific (biomedical) |
| C4         | OpenAI              | 1536           | 500K / 5M       | Web-scale text processing    |
| MSMarco V2 | udever-bloom-1b1    | 1536           | 1M / 10M / 138M | Large-scale search           |

These datasets better simulate today's high-volume, high-dimensional vector data, enabling realistic testing of storage efficiency, query performance, and retrieval accuracy under conditions that match modern AI workloads.


### ⚙️ Custom Dataset Support for Industry-Specific Testing

Every business is unique. The financial industry might need testing focused on transaction embeddings, while social platforms care more about user behavior vectors. VDBBench lets you benchmark with your own data generated from your specific embedding models for your specific workloads.

You can customize:

- Vector dimensions and data types

- Metadata schema and filtering patterns

- Data volume and ingestion patterns

- Query distributions that match your production traffic

After all, no dataset tells a better story than your own production data.


## How VDBBench Measures What Actually Matters in Production

### Production-Focused Metric Design

VDBBench prioritizes metrics that reflect real-world performance, not just lab results. We've redesigned benchmarking around what actually matters in production environments: **reliability under load, tail latency characteristics, sustained throughput, and accuracy preservation.**

- **P95/P99 Latency for Real User Experience**: Average/median latency masks the outliers that frustrate real users and can indicate underlying system instability. VDBBench focuses on tail latency like P95/P99, revealing what performance 95% or 99% of your queries will actually achieve. This is crucial for SLA planning and understanding worst-case user experience.

- **Sustainable Throughput Under Load**: A system that performs well for 5 seconds doesn't cut it in production. VDBBench gradually increases concurrency to find your database's maximum sustainable queries per second (`max_qps`)—not the peak number under short, ideal conditions. This methodology reveals how well your system holds up over time and helps with realistic capacity planning.

- **Recall Balanced with Performance**: Speed without accuracy is meaningless. Every performance number in VDBBench is paired with recall measurements, so you know exactly how much relevance you're trading off for throughput. This enables fair, apples-to-apples comparisons between systems with vastly different internal tradeoffs.


### Test Methodology That Reflects Reality

A key innovation in VDBBench's design is the separation of serial and concurrent testing, which helps capture how systems behave under different types of load and reveals performance characteristics that matter for different use cases.

**Latency Measurement Separation:**

- `serial_latency_p99` measures system performance under minimal load, where only one request is processed at a time. This represents the best-case scenario for latency and helps identify baseline system capabilities.

- `conc_latency_p99` captures system behavior under realistic, high-concurrency conditions, where multiple requests arrive simultaneously and compete for system resources.

**Two-Phase Benchmark Structure**:

1. **Serial Test**: Single-process run of 1,000 queries that establishes baseline performance and accuracy, reporting both `serial_latency_p99` and recall. This phase helps identify the theoretical performance ceiling.

2. **Concurrency Test**: Simulates production environment under sustained load with several key innovations:

   - **Realistic client simulation**: Each test process operates independently with its own connection and query set, avoiding shared-state interference that could distort results

   - **Synchronized start**: All processes begin simultaneously, ensuring measured QPS accurately reflects claimed concurrency levels

   - **Independent query sets**: Prevents unrealistic cache hit rates that don't reflect production query diversity

These carefully structured methods ensure that `max_qps` and `conc_latency_p99` values reported by VDBBench are both accurate and production-relevant, providing meaningful insights for production capacity planning and system design.


## Getting Started with VDBBench 1.0

**VDBBench 1.0** represents a fundamental shift toward production-relevant benchmarking. By covering continuous data writing, metadata filtering with varying selectivity, and streaming loads under concurrent access patterns, it provides the closest approximation to actual production environments available today.

The gap between benchmark results and real-world performance shouldn't be a guessing game. If you're planning to deploy a vector database in production, it's worth understanding how it performs beyond idealized lab tests. VDBBench is open-source, transparent, and designed to support meaningful, apples-to-apples comparisons.

Don't be swayed by impressive numbers that don't translate to production value. **Use VDBBench 1.0 to test scenarios that matter to your business, with your data, under conditions that reflect your actual workload.** The era of misleading benchmarks in vector database evaluation is ending—it's time to make decisions based on production-relevant data.

**Try VDBBench with your own workloads:**[ https://github.com/zilliztech/VectorDBBench](https://github.com/zilliztech/VectorDBBench)

**View testing results of major vector databases:**[ VDBBench Leaderboard](https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch)

Have questions or want to share your results? Join the conversation on[ GitHub](https://github.com/zilliztech/VectorDBBench) or connect with our community on[ Discord](https://discord.com/invite/FG6hMJStWu).


VDBBench 1.0: Real-World Benchmarking for Vector Databases


Announcing VDBBench 1.0: Open-Source Vector Database Benchmarking with Your Real-World Production Workloads



If you've been keeping an eye on the embedding model space, you've probably noticed Alibaba just dropped their [Qwen3 Embedding series](https://qwenlm.github.io/blog/qwen3-embedding/). They released both embedding and reranking models in three sizes each (0.6B, 4B, 8B), all built on the Qwen3 foundation models and designed specifically for retrieval tasks. 

The Qwen3 series has a few features I found interesting:

- **Multilingual embeddings** - they claim a unified semantic space across 100+ languages

- **Instruction prompting** - you can pass custom instructions to modify embedding behavior

- **Variable dimensions** - supports different embedding sizes via Matryoshka Representation Learning

- **32K context length** - can process longer input sequences

- **Standard dual/cross-encoder setup** - embedding model uses dual-encoder, reranker uses cross-encoder

Looking at the benchmarks, Qwen3-Embedding-8B achieved a score of 70.58 on the MTEB multilingual leaderboard, surpassing BGE, E5, and even Google Gemini. The Qwen3-Reranker-8B hit 69.02 on multilingual ranking tasks. These aren't just "pretty good among open-source models" - they're comprehensively matching or even surpassing mainstream commercial APIs. In RAG retrieval, cross-language search, and code search systems, especially in Chinese contexts, these models already have production-ready capabilities.

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdZCKoPqf8mpxwQ_s-gGbdHYvw_HhWn6Ib62v8C_VEZF8AOSnY1yLEEv1ztkINpmwgHAVC5kZw6rWplfx5OkISf_gL4VvoqlXxSfs8s_qd8mdBuA0HBhP9kEdipXy0QVuPmEyOJRg?key=nqzZfIwgkzdlEZQ2MYSMGQ)

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdNppvBpn_5M9d6WDb0-pCjgTobVc9eFw_m6m6Vg73wJtB9OvcPFw5089FUui_N2-LbJVjJPe1c8_EnYY4F3Ryw0021kvmJ0jU0Q06qG2ZX2D1vywIyd5aKqO_cx-77U_spMVr8cQ?key=nqzZfIwgkzdlEZQ2MYSMGQ)

As someone who's likely dealt with the usual suspects (OpenAI's embeddings, BGE, E5), you might be wondering if these are worth your time. Spoiler: they are. 


## What We're Building

This tutorial walks through building a complete RAG system using Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B with Milvus. We'll implement a two-stage retrieval pipeline:

1. **Dense retrieval** with Qwen3 embeddings for fast candidate selection

2. **Reranking** with Qwen3 cross-encoder for precision refinement

3. **Generation** with OpenAI's GPT-4 for final responses

By the end, you'll have a working system that handles multilingual queries, uses instruction prompting for domain tuning, and balances speed with accuracy through intelligent reranking.


## Environment Setup

Let's start with the dependencies. Note the minimum version requirements - they're important for compatibility:

```
pip install --upgrade pymilvus openai requests tqdm sentence-transformers transformers
```


_Requires transformers>=4.51.0 and sentence-transformers>=2.7.0_

For this tutorial, we'll use OpenAI as our generation model. Set up your API key:

```
import os

os.environ["OPENAI_API_KEY"] = "sk-***********"
```



## **Data Preparation**

We'll use Milvus documentation as our knowledge base - it's a good mix of technical content that tests both retrieval and generation quality.

Download and extract the documentation:

```
! wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
! unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
```


Load and chunk the markdown files. We're using a simple header-based splitting strategy here - for production systems, consider more sophisticated chunking approaches:

```
from glob import glob

text_lines = []

for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()

    text_lines += file_text.split("# ")
```



## **Model Setup**

Now let's initialize our models. We're using the lightweight 0.6B versions, which offer a good balance of performance and resource requirements:

```
from openai import OpenAI
from sentence_transformers import SentenceTransformer
import torch
from transformers import AutoModel, AutoTokenizer, AutoModelForCausalLM

# Initialize OpenAI client for LLM generation
openai_client = OpenAI()

# Load Qwen3-Embedding-0.6B model for text embeddings
embedding_model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")

# Load Qwen3-Reranker-0.6B model for reranking
reranker_tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Reranker-0.6B", padding_side='left')
reranker_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-Reranker-0.6B").eval()

# Reranker configuration
token_false_id = reranker_tokenizer.convert_tokens_to_ids("no")
token_true_id = reranker_tokenizer.convert_tokens_to_ids("yes")
max_reranker_length = 8192

prefix = "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n"
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
prefix_tokens = reranker_tokenizer.encode(prefix, add_special_tokens=False)
suffix_tokens = reranker_tokenizer.encode(suffix, add_special_tokens=False)
```


The expected output: 

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdaUrXQrAs2W8-rGT9njJHEKnQ8YwREmULO6xYJnpPy7bwsmZImDRt_3EMwJuVM3k3zI7pbNvY1fDsqMKYq-rrNArx_gxOA4ZTi0g1tkRIlUqJfx1z2nZ60ATPW0L5t6I_XLTVf?key=nqzZfIwgkzdlEZQ2MYSMGQ)


## Embedding Function

The key insight with Qwen3 embeddings is the ability to use different prompts for queries versus documents. This seemingly small detail can significantly improve retrieval performance:

```
def emb_text(text, is_query=False):
    """
    Generate text embeddings using Qwen3-Embedding-0.6B model.
    
    Args:
        text: Input text to embed
        is_query: Whether this is a query (True) or document (False)
    
    Returns:
        List of embedding values
    """
    if is_query:
        # For queries, use the "query" prompt for better retrieval performance
        embeddings = embedding_model.encode([text], prompt_name="query")
    else:
        # For documents, use default encoding
        embeddings = embedding_model.encode([text])
    
    return embeddings[0].tolist()
```


Let's test the embedding function and check the output dimensions:

```
test_embedding = emb_text("This is a test")
embedding_dim = len(test_embedding)
print(f"Embedding dimension: {embedding_dim}")
print(f"First 10 values: {test_embedding[:10]}")
```


Expected output:

```
Embedding dimension: 1024
First 10 values: [-0.009923271834850311, -0.030248118564486504, -0.011494234204292297, ...]
```



## Reranking Implementation

The reranker uses a cross-encoder architecture to evaluate query-document pairs. This is more computationally expensive than the dual-encoder embedding model, but provides much more nuanced relevance scoring.

Here's the complete reranking pipeline:

```
def format_instruction(instruction, query, doc):
    """Format instruction for reranker input"""
    if instruction is None:
        instruction = 'Given a web search query, retrieve relevant passages that answer the query'
    output = "<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}".format(
        instruction=instruction, query=query, doc=doc
    )
    return output

def process_inputs(pairs):
    """Process inputs for reranker"""
    inputs = reranker_tokenizer(
        pairs, padding=False, truncation='longest_first',
        return_attention_mask=False, max_length=max_reranker_length - len(prefix_tokens) - len(suffix_tokens)
    )
    for i, ele in enumerate(inputs['input_ids']):
        inputs['input_ids'][i] = prefix_tokens + ele + suffix_tokens
    inputs = reranker_tokenizer.pad(inputs, padding=True, return_tensors="pt", max_length=max_reranker_length)
    for key in inputs:
        inputs[key] = inputs[key].to(reranker_model.device)
    return inputs

@torch.no_grad()
def compute_logits(inputs, **kwargs):
    """Compute relevance scores using reranker"""
    batch_scores = reranker_model(**inputs).logits[:, -1, :]
    true_vector = batch_scores[:, token_true_id]
    false_vector = batch_scores[:, token_false_id]
    batch_scores = torch.stack([false_vector, true_vector], dim=1)
    batch_scores = torch.nn.functional.log_softmax(batch_scores, dim=1)
    scores = batch_scores[:, 1].exp().tolist()
    return scores

def rerank_documents(query, documents, task_instruction=None):
    """
    Rerank documents based on query relevance using Qwen3-Reranker
    
    Args:
        query: Search query
        documents: List of documents to rerank
        task_instruction: Task instruction for reranking
    
    Returns:
        List of (document, score) tuples sorted by relevance score
    """
    if task_instruction is None:
        task_instruction = 'Given a web search query, retrieve relevant passages that answer the query'
    
    # Format inputs for reranker
    pairs = [format_instruction(task_instruction, query, doc) for doc in documents]
    
    # Process inputs and compute scores
    inputs = process_inputs(pairs)
    scores = compute_logits(inputs)
    
    # Combine documents with scores and sort by score (descending)
    doc_scores = list(zip(documents, scores))
    doc_scores.sort(key=lambda x: x[1], reverse=True)
    
    return doc_scores
```




## Setting Up Milvus Vector Database

Now let's set up our vector database. We're using Milvus Lite for simplicity, but the same code works with full Milvus deployments:

```
from pymilvus import MilvusClient

milvus_client = MilvusClient(uri="./milvus_demo.db")

collection_name = "my_rag_collection"
```


**Deployment Options:**

- **Local file** (like `./milvus.db`): Uses Milvus Lite, perfect for development

- **Docker/Kubernetes**: Use server URI like `http://localhost:19530` for production

- **Zilliz Cloud**: Use cloud endpoint and API key for managed service

Clean up any existing collection and create a new one:

```
# Remove existing collection if it exists
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)

# Create new collection with our embedding dimensions
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim,  # 1024 for Qwen3-Embedding-0.6B
    metric_type="IP",  # Inner product for similarity
    consistency_level="Strong",  # Ensure data consistency
)
```



## Loading Data into Milvus

Now let's process our documents and insert them into the vector database:

```
from tqdm import tqdm

data = []

for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": emb_text(line), "text": line})

milvus_client.insert(collection_name=collection_name, data=data)
```


Expected output:

```
Creating embeddings: 100%|████████████| 72/72 [00:08<00:00, 8.68it/s]
Inserted 72 documents
```



## Enhancing RAG with Reranking Technology

Now comes the exciting part - putting it all together into a complete retrieval-augmented generation system.


### **Step 1: Query and Initial Retrieval**

Let's test with a common question about Milvus:

```
question = "How is data stored in milvus?"

# Perform initial dense retrieval to get top candidates
search_res = milvus_client.search(
    collection_name=collection_name,
    data=[emb_text(question, is_query=True)],  # Use query prompt
    limit=10,  # Get top 10 candidates for reranking
    search_params={"metric_type": "IP", "params": {}},
    output_fields=["text"],  # Return the actual text content
)

print(f"Found {len(search_res[0])} initial candidates")
```



### **Step 2: Reranking for Precision**

Extract candidate documents and apply reranking:

```
# Extract candidate documents
candidate_docs = [res["entity"]["text"] for res in search_res[0]]

# Rerank using Qwen3-Reranker
print("Reranking documents...")
reranked_docs = rerank_documents(question, candidate_docs)

# Select top 3 after reranking
top_reranked_docs = reranked_docs[:3]
print(f"Selected top {len(top_reranked_docs)} documents after reranking")
```




### **Step 3: Compare Results**

Let's examine how reranking changes the results:

```
Reranked results (top 3):
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.9997891783714294
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.9989748001098633
    ],
    [
        "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
        0.9984032511711121
    ]
]

================================================================================
Original embedding-based results (top 3):
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.8306853175163269
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.7302717566490173
    ],
    [
        "How does Milvus handle vector data types and precision?\n\nMilvus supports Binary, Float32, Float16, and BFloat16 vector types.\n\n- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.\n- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.\n- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.\n\n###",
        0.7003671526908875
    ]
]
```


The reranking typically shows much higher discriminative scores (closer to 1.0 for relevant documents) compared to embedding similarity scores.


### **Step 4: Generate Final Response**

Now let's use the retrieved context to generate a comprehensive answer:

First: Convert the retrieved documents to string format.

```
context = "\n".join(
    [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
)
```


Provide system prompt and user prompt for the large language model. This prompt is generated from documents retrieved from Milvus.

```
SYSTEM_PROMPT = """
Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
"""
USER_PROMPT = f"""
Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
<context>
{context}
</context>
<question>
{question}
</question>
"""
```


Use GPT-4o to generate a response based on the prompts.

```
response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
)
print(response.choices[0].message.content)
```


Expected output:

```
In Milvus, data is stored in two main forms: inserted data and metadata. 
Inserted data, which includes vector data, scalar data, and collection-specific 
schema, is stored in persistent storage as incremental logs. Milvus supports 
multiple object storage backends for this purpose, including MinIO, AWS S3, 
Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent 
Cloud Object Storage. Metadata for Milvus is generated by its various modules 
and stored in etcd.
```



## **Wrapping Up**

This tutorial demonstrated a complete RAG implementation using Qwen3's embedding and reranking models. The key takeaways:

1. **Two-stage retrieval** (dense + reranking) consistently improves accuracy over embedding-only approaches

2. **Instruction prompting** allows domain-specific tuning without retraining

3. **Multilingual capabilities** work naturally without additional complexity

4. **Local deployment** is feasible with the 0.6B models

The Qwen3 series offers solid performance in a lightweight, open-source package. While not revolutionary, they provide incremental improvements and useful features like instruction prompting that can make a real difference in production systems.

Test these models against your specific data and use cases - what works best always depends on your content, query patterns, and performance requirements.


Hands-on RAG with Qwen3 Embedding and Reranking Models using Milvus


For the past few years, we've been focused on one thing: building an enterprise-ready vector database for the AI era. The hard part isn't building _a_ database—it's building one that's scalable, easy to use, and actually solves real problems in production.

This June, we reached a new milestone: Milvus hit [35,000 stars on GitHub](https://github.com/milvus-io/milvus) (now it has 35.5K+ stars at the time of writing). We're not going to pretend this is just another number—it means a lot to us.

Each star represents a developer who took the time to look at what we've built, found it useful enough to bookmark, and in many cases, decided to use it. Some of you have gone further: filing issues, contributing code, answering questions in our forums, and helping other developers when they get stuck.

We wanted to take a moment to share our story—the real one, with all the messy parts included.


## We Started Building Milvus Because Nothing Else Worked

Back in 2017, we started with a simple question: As AI applications were starting to emerge and unstructured data was exploding, how do you efficiently store and search the vector embeddings that power semantic understanding?

Traditional databases weren't built for this. They're optimized for rows and columns, not high-dimensional vectors. The existing technologies and tools were either impossible or painfully slow for what we needed.

We tried everything available. Hacked together solutions with Elasticsearch. Built custom indexes on top of MySQL. Even experimented with FAISS, but it was designed as a research library, not a production database infrastructure. Nothing provided the complete solution we envisioned for enterprise AI workloads.

**So we started building our own.** Not because we thought it would be easy—databases are notoriously hard to get right—but because we could see where AI was heading and knew it needed purpose-built infrastructure to get there.

By 2018, we were deep into developing what would become [Milvus](https://milvus.io/). The term "**vector database**" didn't even exist yet. We were essentially creating a new category of infrastructure software, which was both exciting and terrifying.


## Open-Sourcing Milvus: Building in Public

In November 2019, we decided to open-source Milvus version 0.10.

Open-sourcing means exposing all your flaws to the world. Every hack, every TODO comment, every design decision you're not entirely sure about. But we believed that if vector databases were going to become critical infrastructure for AI, they needed to be open and accessible to everyone.

The response was overwhelming. Developers didn't just use Milvus—they improved it. They found bugs we'd missed, suggested features we hadn't considered, and asked questions that made us think harder about our design choices.

In 2020, we joined the [LF AI & Data Foundation](https://lfaidata.foundation/). This wasn't just for credibility—it taught us how to maintain a sustainable open-source project. How to handle governance, backward compatibility, and building software that lasts years, not months.

By 2021, we released Milvus 1.0 and [graduated from LF AI & Data Foundation](https://lfaidata.foundation/projects/milvus/). That same year, we won the [BigANN global challenge](https://big-ann-benchmarks.com/neurips21.html) for billion-scale vector search. That win felt good, but more importantly, it validated that we were solving real problems the right way.


## The Hardest Decision: Starting Over

Here's where things get complicated. By 2021, Milvus 1.0 was working well for many use cases, but enterprise customers kept asking for the same things: better cloud-native architecture, easier horizontal scaling, more operational simplicity.

We had a choice: patch our way forward or rebuild from the ground up. We chose to rebuild.

Milvus 2.0 was essentially a complete rewrite. We introduced a fully decoupled storage-compute architecture with dynamic scalability. It took us two years and was honestly one of the most stressful periods in our company's history. We were throwing away a working system that thousands of people were using to build something unproven.

**But when we released Milvus 2.0 in 2022, it transformed Milvus from a powerful vector database into production-ready infrastructure that could scale to enterprise workloads.** That same year, we also completed a [Series B+ funding round](https://zilliz.com/news/vector-database-company-zilliz-series-b-extension)—not to burn money, but to double down on product quality and support for global customers. We knew this path would take time, but every step had to be built on a solid foundation.


## When Everything Accelerated with AI

2023 was the year of [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) (retrieval-augmented generation). Suddenly, semantic search went from an interesting AI technique to essential infrastructure for chatbots, document Q&A systems, and AI agents.

The GitHub stars of Milvus spiked. Support requests multiplied. Developers who had never heard of vector databases were suddenly asking sophisticated questions about indexing strategies and query optimization.

This growth was exciting but also overwhelming. We realized we needed to scale not just our technology, but our entire approach to community support. We hired more developer advocates, completely rewrote our documentation, and started creating educational content for developers new to vector databases.

We also launched [Zilliz Cloud](https://zilliz.com/cloud)—our fully managed version of Milvus. Some people asked why we were "commercializing" our open-source project. The honest answer is that maintaining enterprise-grade infrastructure is expensive and complex. Zilliz Cloud allows us to sustain and accelerate Milvus development while keeping the core project completely open source.

Then came 2024. [**Forrester named us a leader**](https://zilliz.com/blog/zilliz-named-a-leader-in-the-forrester-wave-vector-database-report) **in the vector database category.** Milvus passed 30,000 GitHub stars. **And we realized: the road we'd been paving for seven years had finally become the highway.** As more enterprises adopted vector databases as critical infrastructure, our business growth accelerated rapidly—validating that the foundation we'd built could scale both technically and commercially.


## The Team Behind Milvus: Zilliz

Here's something interesting: many people know Milvus but not Zilliz. We're actually fine with that. [**Zilliz**](https://zilliz.com/) **is the team behind Milvus—we build it, maintain it, and support it.**

What we care about most are the unglamorous things that make the difference between a cool demo and production-ready infrastructure: performance optimizations, security patches, documentation that actually helps beginners, and responding thoughtfully to GitHub issues.

We've built a 24/7 global support team across the U.S., Europe, and Asia, because developers need help in their time zones, not ours. We have community contributors we call "[Milvus Ambassadors](https://docs.google.com/forms/d/e/1FAIpQLSfkVTYObayOaND8M1ci9eF_YWvoKDb-xQjLJYZ-LhbCdLAt2Q/viewform)" who organize events, answer forum questions, and often explain concepts better than we do.

We've also welcomed integrations with AWS, GCP, and other cloud providers—even when they offer their own managed versions of Milvus. More deployment options are good for users. Though we've noticed that when teams hit complex technical challenges, they often end up reaching out to us directly because we understand the system at the deepest level.

Many people think open source is just a "toolbox," but it's actually an "evolutionary process"—a collective effort by countless people who love and believe in it. Only those who truly understand the architecture can provide the "why" behind bug fixes, performance bottleneck analysis, data system integration, and architectural adjustments.

**So if you're using open-source Milvus, or considering vector databases as a core component of your AI system, we encourage you to reach out to us directly for the most professional and timely support.**


## Real Impact in Production: The Trust from Users

The use cases for Milvus have grown beyond what we initially imagined. We're powering AI infrastructure for some of the world's most demanding enterprises across every industry.

![zilliz customers.png](https://assets.zilliz.com/zilliz_customers_e7340d5dd4.png)


[**Bosch**](https://zilliz.com/customers/bosch), the global automotive technology leader and pioneer in autonomous driving, revolutionized their data analysis with Milvus achieving 80% reduction in data collection costs and $1.4M annual savings while searching billions of driving scenarios in milliseconds for critical edge cases.

[**Read AI**](https://zilliz.com/customers/read-ai), one of the fastest-growing productivity AI companies serving millions of monthly active users, uses Milvus to achieve sub-20-50ms retrieval latency across billions of records and 5× speedup in agentic search. Their CTO says, "Milvus serves as the central repository and powers our information retrieval among billions of records."

[**A global fintech leader**](https://zilliz.com/customers/global-fintech-leader), one of the world's largest digital payment platforms processing tens of billions of transactions across 200+ countries and 25+ currencies, chose Milvus for 5-10× faster batch ingestion than competitors, completing jobs in under 1 hour that took others 8+ hours.

[**Filevine**](https://zilliz.com/customers/filevine), the leading legal work platform trusted by thousands of law firms across the United States, manages 3 billion vectors across millions of legal documents, saving attorneys 60-80% of time in document analysis and achieving "true consciousness of data" for legal case management.

We're also supporting **NVIDIA, OpenAI, Microsoft, Salesforce, Walmart,** and many others in almost every industry. Over 10,000 organizations have made Milvus or Zilliz Cloud their vector database of choice.

These aren't just technical success stories—they're examples of how vector databases are quietly becoming critical infrastructure that powers the AI applications people use every day.


## Why We Built Zilliz Cloud: Enterprise-Grade Vector Database as a Service

Milvus is open-source and free to use. But running Milvus well at enterprise scale requires deep expertise and significant resources. Index selection, memory management, scaling strategies, security configurations—these aren't trivial decisions. Many teams want the power of Milvus without the operational complexity and with enterprise support, SLA guarantees, etc.

That's why we built [Zilliz Cloud](https://zilliz.com/cloud)—a fully managed version of Milvus deployed across 25 global regions and 5 major clouds, including AWS, GCP, and Azure, designed specifically for enterprise-scale AI workloads that demand performance, security, and reliability.

Here's what makes Zilliz Cloud different:

- **Massive Scale with High Performance:** Our proprietary AI-powered AutoIndex engine delivers 3-5× faster query speeds than open-source Milvus, with zero index tuning required. The cloud-native architecture supports billions of vectors and tens of thousands of concurrent queries while maintaining sub-second response times.

- [**Built-in Security & Compliance**](https://zilliz.com/trust-center)**:** Encryption at rest and in transit, fine-grained RBAC, comprehensive audit logging, SAML/OAuth2.0 integration, and [BYOC](https://zilliz.com/bring-your-own-cloud) (bring your own cloud) deployments. We're compliant with GDPR, HIPAA, and other global standards that enterprises actually need. 

- **Optimized for Cost-Efficiency:** Tiered hot/cold data storage, elastic scaling that responds to real workloads, and pay-as-you-go pricing can reduce total cost of ownership by 50% or more compared to self-managed deployments.

- **Truly Cloud-Agnostic without vendor lock-in:** Deploy on AWS, Azure, GCP, Alibaba Cloud, or Tencent Cloud without vendor lock-in. We ensure global consistency and scalability regardless of where you run.

These capabilities might not sound flashy, but they solve real, daily problems that enterprise teams face when building AI applications at scale. And most importantly: it's still Milvus under the hood, so there's no proprietary lock-in or compatibility issues.


## What's Next: Vector Data Lake

We coined the term "[vector database](https://zilliz.com/learn/what-is-vector-database)" and were the first to build one, but we're not stopping there. We're now building the next evolution: **Vector Data Lake.**

**Here's the problem we're solving: not every vector search needs millisecond latency.** Many enterprises have massive datasets that are queried occasionally, including historical document analysis, batch similarity computations, and long-term trend analysis. For these use cases, a traditional real-time vector database is both overkill and expensive. 

Vector Data Lake uses a storage-compute separated architecture specifically optimized for massive-scale, infrequently accessed vectors while keeping costs dramatically lower than real-time systems.

**Core capabilities include:**

- **Unified Data Stack:** Seamlessly connects online and offline data layers with consistent formats and efficient storage, so you can move data between hot and cold tiers without reformatting or complex migrations.

- **Compatible Compute Ecosystem:** Works natively with frameworks like Spark and Ray, supporting everything from vector search to traditional ETL and analytics. This means your existing data teams can work with vector data using tools they already know.

- **Cost-Optimized Architecture:** Hot data stays on SSD or NVMe for fast access; cold data automatically moves to object storage like S3. Smart indexing and storage strategies keep I/O fast when you need it while making storage costs predictable and affordable.

This isn't about replacing vector databases—it's about giving enterprises the right tool for each workload. Real-time search for user-facing applications, cost-effective vector data lakes for analytics and historical processing.

We still believe in the logic behind Moore's Law and Jevons Paradox: as the unit cost of computing drops, adoption scales. The same applies to vector infrastructure.

By improving indexes, storage structures, caching, and deployment models—day in, day out—we hope to make AI infrastructure more accessible and affordable for everyone, and to help bring unstructured data into the AI-native future.


## A Big Thanks to You All!

Those 35K+ stars represent something we're genuinely proud of: a community of developers who find Milvus useful enough to recommend and contribute to.

But we're not done. Milvus has bugs to fix, performance improvements to make, and features our community has been asking for. Our roadmap is public, and we genuinely want your input on what to prioritize.

The number itself isn't what matters—it's the trust those stars represent. Trust that we'll keep building in the open, keep listening to feedback, and keep making Milvus better.

- **To our contributors:** your PRs, bug reports, and documentation improvements make Milvus better every day. Thank you so much. 

- **To our users:** thank you for trusting us with your production workloads and for the feedback that keeps us honest.

- **To our community:** thank you for answering questions, organizing events, and helping newcomers get started.

If you're new to vector databases, we'd love to help you get started. If you're already using Milvus or Zilliz Cloud, we'd love to [hear about your experience](https://zilliz.com/share-your-story). And if you're just curious about what we're building, our community channels are always open.

Let's keep building the infrastructure that makes AI applications possible—together.

***

Find us here: [Milvus on GitHub](https://github.com/milvus-io/milvus) |[ Zilliz Cloud](https://zilliz.com/) |[ Discord](https://discuss.milvus.io/) | [LinkedIn](https://www.linkedin.com/company/the-milvus-project/) | [X](https://x.com/zilliz_universe) | [YouTube](https://www.youtube.com/@MilvusVectorDatabase/featured)

[![](https://assets.zilliz.com/office_hour_4fb9130a9b.png)](https://meetings.hubspot.com/chloe-williams1/milvus-office-hour?__hstc=175614333.dc4bcf53f6c7d650ea8978dcdb9e7009.1727350436713.1751017913702.1751029841530.667&__hssc=175614333.3.1751029841530&__hsfp=3554976067)







Our Journey to 35K+ GitHub Stars: Building Milvus from Scratch


Our Journey to 35K+ GitHub Stars: The Real Story of Building Milvus from Scratch



Every company has the same problem: valuable knowledge trapped in PDFs, Word docs, and file shares that nobody can find when they need it. Support teams answer the same questions repeatedly, while developers waste hours searching through outdated documentation.

**What if your documents could answer questions directly?**

This tutorial shows you how to build a production-ready AI assistant that:

- Transforms your static documents into an intelligent Q&A system

- Maintains conversation context and memory

- Scales to handle enterprise workloads

- Includes security, monitoring, and observability out of the box


## What We'll Build

By the end of this tutorial, you'll have:

- A document ingestion pipeline that processes PDFs and Word docs

- A vector search system powered by Milvus for semantic search

- A chat API with memory and context awareness

- Enterprise-grade security and monitoring

- A complete working example you can deploy


## Key Components We’ll Use

- [**Spring Boot**](https://github.com/spring-projects/spring-boot) is a widely used Java framework for building backend applications with minimal configuration. It offers strong developer productivity, seamless integration with modern tooling, and built-in support for REST APIs, observability, and security.

- [**Milvus**](https://milvus.io/) is an open-source, high-performance, cloud-native vector database designed for semantic search. It allows you to store and search embeddings with millisecond-scale latency, even across billions of vectors.

- [**RAG**](https://zilliz.com/learn/Retrieval-Augmented-Generation) is an architecture that combines retrieval and generation: it fetches relevant knowledge snippets from a vector database like Milvus, then uses a language model to craft a fluent, contextual response.

- [**Ollama**](https://ollama.com/): Local AI model provider (OpenAI-compatible, completely free)


## Prerequisites

Before starting, ensure you have:

- Java 17+ installed

- Docker, Docker Compose

- Git for cloning the example repository

- Ollama installed and running locally

- Milvus (via Docker)

- Spring Boot 3.5.0 + Spring AI 1.0.0

- Micrometer, Testcontainers


## Environment Setup

Clone the Example Repository/: <https://github.com/topikachu/spring-ai-rag>


```
git clone https://github.com/topikachu/spring-ai-rag
cd spring-ai-rag
```
Verify your environment: 
```
# Verify Docker is running correctly
docker version
docker ps

# Verify Java version
java -version

# Verify Ollama installation
ollama --version
```

Download Ollama Models: 
```
# Pull required models for this project
ollama pull mistral          # Chat model
ollama pull nomic-embed-text # Embedding model

# Verify models are available
ollama list
```

Key Configuration (application.properties)
```
# Ollama Configuration (OpenAI-compatible API)
spring.ai.openai.base-url=http://localhost:11434
spring.ai.openai.chat.options.model=mistral
spring.ai.openai.embedding.options.model=nomic-embed-text
spring.ai.openai.embedding.options.dimensions=768

# Vector Store Configuration - dimensions must match embedding model
spring.ai.vectorstore.milvus.embedding-dimension=768
```


## Document ETL: Structuring Unstructured Text

This section walks through the heart of the system---transforming unstructured files into searchable, intelligent responses using vector embeddings, Milvus indexing, and Spring AI's RAG pipeline.

**Workflow Overview:**

- Use `TikaDocReader` to read PDFs and Word files

- Use token-based splitting to chunk documents while preserving context

- Generate embeddings using the OpenAI-compatible embedding model

- Store the embeddings in Milvus for later semantic search

![](https://assets.zilliz.com/workflow_7e9f990b18.png )




Sample Implementation

```
public Flux<Document> ingestionFlux() {
  return documentReader.getDocuments()
          .flatMap(document -> {
            var processChunks = Mono.fromRunnable(() -> {
              var chunks = textSplitter.apply(List.of(document));
              vectorStore.write(chunks); // expensive operation
            }).subscribeOn(Schedulers.boundedElastic());

            return Flux.concat(
                    Flux.just(document),
                    processChunks.then(Mono.empty())
            );
          })
          .doOnComplete(() -> log.info("RunIngestion() finished"))
          .doOnError(e -> log.error("Error during ingestion", e));
}
```


## Vector Storage: Millisecond-Scale Semantic Search with Milvus

Configuration Example: 

```
spring.ai.vectorstore.milvus.initialize-schema=true
spring.ai.vectorstore.milvus.embedding-dimension=768
```



📌 **Example:** When a user asks "Does Spring Boot support reactive programming with WebFlux?", Milvus returns related documentation segments, and the AI model generates a natural language answer with specific implementation details.


## Building a RAG-Enabled Chat: Contextual Q&A with Memory Integration

Core workflow: 

1. The user submits a question

2. Vector search retrieves the most relevant document chunks

3. The system loads past conversation context (via Redis)

4. The AI model generates a response that includes both new and historical context


![](https://assets.zilliz.com/build_a_rag_chat_workflow_976dcd9aa2.png)








Retrieval + Memory Chat Integration Example: 

```
public ChatClient.ChatClientRequestSpec input(String userInput, String conversationId) {
  return chatClient.prompt()
          .advisors(
                  messageChatMemoryAdvisor,
                  retrievalAugmentationAdvisor
          )
          .advisors(spec -> spec.param(CONVERSATION_ID, conversationId))
          .user(userInput);
}
```


For a smoother frontend experience, use the reactive stream API to return `Flux` content via server-sent events (SSE)---ideal for "typing" effects:

```
public Flux<String> stream(String userInput, String conversationId) {
    return input(userInput, conversationId)
            .stream().content();
}
```


REST API Controller:

```
@PostMapping(path = "/chat", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> chat(@RequestBody ChatRequest chatRequest, @RequestParam() String conversationId, Principal principal) {
  var conversationKey = String.format("%s:%s", principal.getName(), conversationId);
  return chatService.stream(chatRequest.userInput, conversationKey)
          .doOnError(exp -> log.error("Error in chat", exp));
}
```



## Enterprise-Grade API Security and System Observability

This section ensures your AI assistant doesn't just work---it runs securely, is traceable, and performs under real-world workloads.


### API Security: Role-Based Access Control

**Example: Securing Admin Endpoints**

```
@Override
protected void configure(HttpSecurity http) throws Exception {
    http
        .httpBasic()
        .and()
        .authorizeRequests(authz -> authz
            .antMatchers("/api/v1/index").hasRole("ADMIN")
            .anyRequest().authenticated()
        );
}
```


💡 **Production Tip:** For real-world deployments, use OAuth2 or JWT for scalable authentication.


### Observability: Full Stack Tracing and Metrics

**Tracing:** We'll use OpenTelemetry JavaAgent to trace full request flows from user chat to Milvus search and LLM response---including gRPC spans:

```
-javaagent:<path/to/opentelemetry-javaagent.jar> \
-Dotel.metrics.exporter=none \
-Dotel.logs.exporter=none
```


**Metrics:** Micrometer automatically exposes Prometheus-friendly metrics:

- Model Response Time

```
# HELP gen_ai_client_operation_seconds  
# TYPE gen_ai_client_operation_seconds summary
gen_ai_client_operation_seconds_count{...} 1
```


- Vector Retrieval Time

```
# HELP db_vector_client_operation_seconds
# TYPE db_vector_client_operation_seconds summary
db_vector_client_operation_seconds_count{...} 1
```


Configuration: 

```
management.endpoints.web.exposure.include=prometheus
```


💡 **Technical Note:** Spring Boot 3.2 introduces OTEL starters, but they don't cover gRPC (used by Milvus). To ensure end-to-end visibility, this project uses the JavaAgent approach.


## Running the Project: End-to-End Execution

Start the Complete System


```
export OPENAI_API_KEY=dummy
export SPRING_PROFILES_ACTIVE=ollama-openai
ollama pull mistral            # Pull chat model
ollama pull nomic-embed-text   # Pull embedding model

mvn clean test package
docker compose up -d
java -javaagent:target/otel/opentelemetry-javaagent.jar -Dotel.metrics.exporter=none -Dotel.logs.exporter=none  -Dinput.directory=$PWD/src/test/resources/corpus  -jar target/rag-0.0.1-SNAPSHOT.jar

curl --location 'localhost:8080/api/v1/index' \
--user "admin:password" \
--header 'Content-Type: application/json' \
--data '{}'

curl --location 'localhost:8080/api/v1/chat?conversationId=flat' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "Does milvus support FLAT type index?"
}'

curl --location 'localhost:8080/api/v1/chat?conversationId=flat' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "When shall I use this index type?"
}'

curl --location 'localhost:8080/api/v1/chat?conversationId=hnsw' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "Does milvus support HNSW type index?"
}'

curl --location 'localhost:8080/api/v1/chat?conversationId=hnsw' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "When shall I use this index type?"
}'

curl "http://localhost:8080/actuator/prometheus"
```


To view tracing UI, open[ http://localhost:16686/](http://localhost:16686/)

![](https://assets.zilliz.com/To_view_tracing_UI_686e8f54b9.png)


## Conclusion

You now have a production-ready AI assistant that transforms static documents into intelligent conversations. The system includes:

✅ **Document Processing**: Automated ingestion and vectorization
✅ **Semantic Search**: Fast, accurate retrieval with Milvus
✅ **Conversation Memory**: Context-aware chat experiences
✅ **Enterprise Security**: Authentication and access control 

✅ **Full Observability**: Monitoring, tracing, and metrics

By combining Spring Boot, Milvus, and Ollama, we turn static enterprise documents into dynamic, context-aware conversations---with full observability, memory, and security built-in.

Whether you're building internal copilots, domain-specific assistants, or customer-facing support bots, this architecture is designed to scale your workload and keep you in control of your data.

Curious about what Milvus can do for your AI stack? Explore the[ Milvus open-source project](https://milvus.io), try[ managed Milvus (Zilliz Cloud](https://zilliz.com)) for a hassle-free experience, or join our [Discord channel](https://discord.com/invite/8uyFbECzPX) for more hands-on guides like this.


Building a Production-Ready AI Assistant with Spring Boot and Milvus


From Docs to Dialogue: Building a Production-Ready AI Assistant with Spring Boot and Milvus




## The One Thing Breaking Your Vibe Coding Flow

Vibe Coding is having its moment. Tools like Cursor and Windsurf are redefining how we write software, making development feel effortless and intuitive. Ask for a function and get a snippet. Need a quick API call? It's generated before you finish typing.

**However, here's the catch that's ruining the vibe: AI assistants often generate outdated code that breaks in production.** This is because LLMs powering these tools often rely on outdated training data. Even the slickest AI copilot can suggest code that’s a year—or three—behind the curve. You might end up with a syntax that no longer works, deprecated API calls, or practices that today’s frameworks actively discourage. 

Consider this example: I asked Cursor to generate Milvus connection code, and it produced this:


```
connections.connect("default", host="localhost", port="19530")
```

This used to work perfectly, but the current pymilvus SDK recommends using `MilvusClient` for all connections and operations. The old method is no longer considered best practice, yet AI assistants continue to suggest it because their training data is often months or years out of date.

Even worse, when I requested OpenAI API code, Cursor generated a snippet using `gpt-3.5-turbo`—a model now marked _Legacy_ by OpenAI, costing triple the price of its successor while delivering inferior results. The code also relied on `openai.ChatCompletion`, an API deprecated as of March 2024.

​​![](https://assets.zilliz.com/gpt_pricing_6bfa92d83b.png)

This isn't just about broken code—it's about **broken flow**. The whole promise of Vibe Coding is that development should feel smooth and intuitive. But when your AI assistant generates deprecated APIs and outdated patterns, the vibe dies. You're back to Stack Overflow, back to documentation hunting, back to the old way of doing things.

Despite all the progress in Vibe Coding tools, developers still spend significant time bridging the "last mile" between generated code and production-ready solutions. The vibe is there, but the accuracy isn't.

**Until now.**


## Meet Milvus MCP: Vibe Coding with Always-Up-to-Date Docs

So, is there a way to combine the powerful codegen of tools like Cursor _with_ fresh documentation, so we can generate accurate code right inside the IDE?

Absolutely. By combining the Model Context Protocol (MCP) with Retrieval-Augmented Generation (RAG), we’ve created an enhanced solution called **Milvus MCP**. It helps developers using the Milvus SDK to automatically access the latest docs, enabling their IDE to produce the correct code. This service will be available soon—here’s a sneak peek at the architecture behind it.


### How It Works

![](https://assets.zilliz.com/The_Architecture_Behind_MCP_c9093162b6.png)

The diagram above shows a hybrid system that combines MCP (Model Context Protocol) and RAG (Retrieval-Augmented Generation) architectures to help developers generate accurate code. 

On the left side, developers working in AI-powered IDEs like Cursor or Windsurf interact through a chat interface, which triggers MCP tool calls. These requests are sent to the MCP Server on the right side, which hosts specialized tools for everyday coding tasks like code generation and refactoring.

The RAG component operates on the MCP server side, where the Milvus documentation has been pre-processed and stored as vectors in a Milvus database. When a tool receives a query, it performs a semantic search to retrieve the most relevant documentation snippets and code examples. This contextual information is then sent back to the client, where an LLM uses it to generate accurate, up-to-date code suggestions. 


### MCP transport mechanism 

MCP supports two transport mechanisms: `stdio` and `SSE`:

- Standard Input/Output (stdio): The `stdio` transport allows communication over standard input/output streams. It’s particularly useful for local tools or command-line integrations.

- Server-Sent Events (SSE): SSE supports server-to-client streaming using HTTP POST requests for client-to-server communication.

Because `stdio` relies on local infrastructure, users must manage document ingestion themselves. In our case, **SSE is a better fit**—the server handles all document processing and updates automatically. For example, docs can be re-indexed daily. Users only need to add this JSON config to their MCP setup:

```
{
  "mcpServers": {
    "milvus-code-generate-helper": {
      "url": "http://<SERVER_ADDRESS>:23333/milvus-code-helper/sse"
    }
  }
}
```


Once this is in place, your IDE (such as Cursor or Windsurf) can start communicating with the server-side tools—automatically retrieving the latest Milvus documentation for smarter, up-to-date code generation.


## Milvus MCP in Action

To show how this system works in practice, we've created three ready-to-use tools on the Milvus MCP Server that you can access directly from your IDE. Each tool solves a common problem developers face when working with Milvus:

- **pymilvus-code-generator**: Writes Python code for you when you need to perform common Milvus operations like creating collections, inserting data, or running searches using the pymilvus SDK.

- **orm-client-code-convertor**: Modernizes your existing Python code by replacing outdated ORM (Object Relational Mapping) patterns with the simpler, newer MilvusClient syntax.

- **language-translator**: Converts your Milvus SDK code between programming languages. For instance, if you have working Python SDK code but need it in TypeScript SDK, this tool translates it for you.

Now, let’s take a look at how they work. 


### pymilvus-code-generator


<div style="padding:66.98% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1093504910?badge=0&amp;autopause=0&amp;player_id=0&amp;app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="pymilvus-code-generator"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>



In this demo, I asked Cursor to generate full-text search code using `pymilvus`. Cursor successfully invokes the correct MCP tool and outputs spec-compliant code. Most `pymilvus` use cases work seamlessly with this tool.

Here’s a side-by-side comparison with and without this tool. 

**With MCP MCP:**

![](https://assets.zilliz.com/With_Milvus_MCP_f72ad4cfb6.png)

→ Cursor with Milvus MCP uses the latest `MilvusClient` interface to create a collection.

**Without MCP:**

![](https://assets.zilliz.com/Without_Milvus_MCP_3336d956a4.png)

→ The Cursor without the Milvus MCP server uses outdated ORM syntax—no longer advised.


### orm-client-code-convertor


<div style="padding:66.98% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1093504859?badge=0&amp;autopause=0&amp;player_id=0&amp;app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="orm-client-code-convertor"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>


In this example, the user highlights some ORM-style code and requests a conversion. The tool correctly rewrites the connection and schema logic using a `MilvusClient` instance. The user can accept all changes with one click.


### **language-translator**

<div style="padding:66.98% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1093504885?badge=0&amp;autopause=0&amp;player_id=0&amp;app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="tool3 ts-1"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>

Here, the user selects a `.py` file and asks for a TypeScript translation. The tool calls the correct MCP endpoint, retrieves the latest TypeScript SDK docs, and outputs an equivalent `.ts` file with the same business logic. This is ideal for cross-language migrations.


## Comparing Milvus MCP with Context7, DeepWiki, and Other Tools

We’ve discussed the “last mile” hallucination problem in Vibe Coding. Beyond our Milvus MCP, many other tools also aim to solve this issue, such as Context7 and DeepWiki. These tools, often powered by MCP or RAG, help inject up-to-date docs and code samples into the model’s context window. 


### Context7

![](https://assets.zilliz.com/Context7_fc32b53a0e.png)

Figure: Context7’s Milvus page lets users search and customize doc snippets (<https://context7.com/milvus-io/milvus>)

Context7 provides up-to-date, version-specific documentation and code examples for LLMs and AI code editors. The core problem it addresses is that LLMs rely on outdated or generic information about the libraries you use, giving you code examples that are outdated and based on year-old training data.

Context7 MCP pulls up-to-date, version-specific documentation and code examples straight from the source and places them directly into your prompt. It supports GitHub repo imports and `llms.txt` files, including formats like `.md`, `.mdx`, `.txt`, `.rst`, and `.ipynb` (not `.py` files).

Users can either manually copy content from the site or use Context7’s MCP integration for automated retrieval.


### **DeepWiki**

![](https://assets.zilliz.com/Deep_Wiki_bebe01aa6f.png)

Figure: DeepWiki provides auto-generated summaries of Milvus, including logic and architecture (<https://deepwiki.com/milvus-io/milvus>)

DeepWiki auto-parses open-source GitHub projects to create readable technical docs, diagrams, and flowcharts. It includes a chat interface for natural language Q&A. However, it prioritizes code files over documentation, so it may overlook key doc insights. It currently lacks MCP integration.


### Cursor Agent Mode

![](https://assets.zilliz.com/Cursor_Agent_Mode_fba8ef66af.png)

Agent mode in Cursor enables web search, MCP calls, and plugin toggles. While powerful, it’s sometimes inconsistent. You can use `@` to manually insert docs, but that requires you to find and attach the content first.


### llms.txt

`llms.txt` isn’t a tool—it’s a proposed standard to provide LLMs with structured website content. Usually, in Markdown, it goes in a site’s root directory and organizes titles, doc trees, tutorials, API links, and more.

It’s not a tool on its own, but it pairs well with those that support it.


### Side-by-Side Feature Comparison: Milvus MCP vs. Context7 vs. DeepWiki vs Cursor Agent Mode vs llms.txt 

|                       |                          |                             |                         |                         |                                 |
| :-------------------: | :----------------------: | :-------------------------: | :---------------------: | :---------------------: | :-----------------------------: |
|      **Feature**      |       **Context7**       |         **DeepWiki**        |  **Cursor Agent Mode**  |       **llms.txt**      |          **Milvus MCP**         |
|    **Doc Handling**   |    Docs only, no code    | Code-focused, may miss docs |      User-selected      |   Structured Markdown   |    Official Milvus docs only    |
| **Context Retrieval** |        Auto-inject       |      Manual copy/paste      |   Mixed, less accurate  | Structured pre-labeling | Auto-retrieve from vector store |
|   **Custom Import**   |    ✅ GitHub, llms.txt    |   ✅ GitHub (incl. private)  | ❌ Manual selection only |   ✅ Manually authored   |       ❌ Server-maintained       |
|   **Manual Effort**   | Partial (MCP vs. manual) |         Manual copy         |       Semi-manual       |        Admin only       |      No user action needed      |
|  **MCP Integration**  |           ✅ Yes          |             ❌ No            |    ✅ Yes (with setup)   |       ❌ Not a tool      |            ✅ Required           |
|     **Advantages**    |  Live updates, IDE-ready | Visual diagrams, QA support |     Custom workflows    |  Structured data for AI |   Maintained by Milvus/Zilliz   |
|    **Limitations**    |   No code file support   |          Skips docs         |  Relies on web accuracy |   Requires other tools  |     Focused solely on Milvus    |

Milvus MCP is built specifically for Milvus database development. It automatically gets the latest official documentation and works seamlessly with your coding environment. If you're working with Milvus, this is your best option.

Other tools like Context7, DeepWiki, and Cursor Agent Mode work with many different technologies, but they're not as specialized or accurate for Milvus-specific work.

Choose based on what you need. The good news is these tools work well together - you can use several at once to get the best results for different parts of your project.


## Milvus MCP is Coming Soon!

The hallucination problem in Vibe Coding isn't just a minor inconvenience—it's a productivity killer that forces developers back into manual verification workflows. Milvus MCP demonstrates how specialized MCP servers can solve this by providing real-time access to current documentation.

For Milvus developers, this means no more debugging deprecated `connections.connect()` calls or wrestling with outdated ORM patterns. The three tools—pymilvus-code-generator, orm-client-code-convertor, and language-translator—handle the most common pain points automatically.

Ready to try it? The service will be available soon for early access testing. Stay tuned. 


Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP



As AI-powered search has evolved from experimental projects to mission-critical infrastructure, the demands on [vector databases](https://milvus.io/blog/what-is-a-vector-database.md) have intensified. Organizations need to handle billions of vectors while managing infrastructure costs, supporting real-time data ingestion, and providing sophisticated retrieval beyond basic [similarity search](https://zilliz.com/learn/vector-similarity-search). To tackle these evolving challenges, we've been hard at work developing and refining Milvus. The community response has been incredibly encouraging, with valuable feedback helping shape our direction. 

After months of intensive development, we're excited to announce that **Milvus 2.6 is now available**. This release directly addresses the most pressing challenges in vector search today: **_scaling efficiently while keeping costs under control._**

Milvus 2.6 delivers breakthrough innovations across three critical areas: **cost reduction, advanced search capabilities, and architectural improvements for massive scale**. The results speak for themselves:

- **72% memory reduction** with RaBitQ 1-bit quantization while delivering 4x faster queries

- **50% cost savings** through intelligent tiered storage

- **4x faster full-text search** than Elasticsearch with our enhanced BM25 implementation

- **100x faster** JSON filtering with the newly introduced Path Index

- **Search freshness is achieved economically** with the new zero-disk architecture 

- **Streamlined embedding workflow** with the new “data in and data out” experience 

- **Up to 100K collections in a single cluster** for future-proof multi-tenancy


## Innovations for Cost Reduction: Making Vector Search Affordable

Memory consumption presents one of the biggest challenges when scaling vector search to billions of records. Milvus 2.6 introduces several key optimizations that significantly reduce your infrastructure costs while also improving performance.


### RaBitQ 1-bit Quantization: 72% Memory Reduction with 4× Performance

Traditional quantization methods force you to trade in search quality for memory savings. Milvus 2.6 changes this with [RaBitQ 1-bit quantization](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md) combined with an intelligent refinement mechanism.

The new IVF_RABITQ index compresses the main index to 1/32 of its original size through 1-bit quantization. When used together with an optional SQ8 refinement, this approach maintains high search quality (95% recall) using only 1/4 of the original memory footprint. 

Our preliminary benchmarks reveal promising results: 

|  **Performance Metric** | **Traditional IVF_FLAT** | **RaBitQ (1-bit) Only** | **RaBitQ (1-bit) + SQ8 Refine** |
| --------------------- | ----------------------- | --------------------- | ----------------------------- |
|     Memory Footprint    |      100% (baseline)      |    3% (97% reduction)   |       28% (72% reduction)       |
|          Recall         |           95.2%           |          76.3%          |              94.9%              |
| Search Throughput (QPS) |            236            |    648 (2.7× faster)    |         946 (4× faster)         |

_Table: VectorDBBench evaluation with 1M vectors of 768 dimensions, tested on AWS m6id.2xlarge_

The real breakthrough here isn't just the 72% memory reduction, but achieving this while simultaneously delivering a 4× throughput improvement. This means you can serve the same workload with 75% fewer servers or handle 4× more traffic on your existing infrastructure, all without sacrificing the recall. 

For enterprise users utilizing fully managed Milvus on[ Zilliz Cloud](https://zilliz.com/cloud), we’re developing an automated strategy that dynamically adjusts RaBitQ parameters based on your specific workload characteristics and precision requirements. You will simply enjoy greater cost-effectiveness across all Zilliz Cloud CU types.


### Hot-Cold Tiered Storage: 50% Cost Reduction Through Intelligent Data Placement

Real-world vector search workloads contain data with vastly different access patterns. Frequently accessed data needs instant availability, while archival data can tolerate slightly higher latency in exchange for dramatically lower storage costs.

Milvus 2.6 introduces a tiered storage architecture that automatically classifies data based on access patterns and places it in appropriate storage tiers:

- **Intelligent data classification**: Milvus automatically identifies hot (frequently accessed) and cold (rarely accessed) data segments based on access patterns

- **Optimized storage placement**: Hot data remains in high-performance memory/SSD, while cold data moves to more economical object storage

- **Dynamic data movement**: As usage patterns change, data automatically migrates between tiers

- **Transparent retrieval**: When queries touch cold data, it's automatically loaded on demand

The result is up to 50% reduction in storage costs while maintaining query performance for active data.


### Additional Cost Optimizations

Milvus 2.6 also introduces Int8 vector support for HNSW indexes, Storage v2 format for optimized structure that reduces IOPS and memory requirements, and easier installation directly through APT/YUM package managers.


## Advanced Search Capabilities: Beyond Basic Vector Similarity

Vector search alone isn't enough for modern AI applications. Users demand the precision of traditional information retrieval combined with the semantic understanding of vector embeddings. Milvus 2.6 introduces a suite of advanced search features that bridge this gap.


### Turbocharged BM25: 400% Faster Full-Text Search Than Elasticsearch

[Full-text search](https://milvus.io/blog/full-text-search-in-milvus-what-is-under-the-hood.md) has become essential for building hybrid retrieval systems in vector databases. In Milvus 2.6, significant performance improvements have been made to full-text search, building on the BM25 implementation introduced since version 2.5. For example, this release introduces new parameters like `drop_ratio_search` and `dim_max_score_ratio`, enhancing precision and speed tuning and offering more fine-grained search controls. 

Our benchmarks against the industry-standard BEIR dataset show Milvus 2.6 achieving 3-4× higher throughput than Elasticsearch with equivalent recall rates. For specific workloads, the improvement reaches 7× higher QPS.

![](https://assets.zilliz.com/milvus_vs_ES_when_QPS_with_top_K1000_cadd1ac921.png)


### JSON Path Index: 100x Faster Filtering

Milvus has supported JSON data type for a long time, but filtering on JSON fields was slow due to the lack of index support. Milvus 2.6 adds support for JSON path index to boost the performance significantly. 

Consider a user profile database where each record contains nested metadata like:

```
{
  "user": {
    "location": {
      "city": "San Francisco",
      "country": "USA"
    },
    "interests": ["AI", "Databases", "Cloud Computing"]
  },
  "subscription": {
    "plan": "enterprise",
    "status": "active"
  }
}
```


For a semantic search “users interested in AI” scoped to only San Francisco, Milvus used to parse and evaluate the entire JSON object for every record, making the query very expensive and slow.

Now, Milvus allows you to create indexes on specific paths within JSON fields to speed up the search:

```
index_params.add_index(
    field_name="metadata",
    index_type="INVERTED",
    index_name="json_index",
    params={
        "json_path": "metadata[\"user\"][\"location\"][\"city\"]",  
        "json_cast_type": "varchar"
    }
```

In our performance testing with 100M+ records, JSON Path Index reduced filter latency from **140ms** (P99: 480ms) to just **1.5ms** (P99: 10ms)—a 99% latency reduction that makes such searches practical in production.

This feature is particularly valuable for: 

- Recommendation systems with complex user attribute filtering

- RAG applications that filter documents by metadata

- Multi-tenant systems where data segmentation is critical


### Enhanced Text Processing and Time-Aware Search

Milvus 2.6 introduces a completely revamped text analysis pipeline with sophisticated language handling, including the Lindera tokenizer for Japanese and Korean, the ICU tokenizer for comprehensive multilingual support, and enhanced Jieba with custom dictionary integration.

**Phrase Match Intelligence** captures semantic nuance in word order, distinguishing between "machine learning techniques" and "learning machine techniques":

```
PHRASE_MATCH(document_text, "artificial intelligence research", slop=1)
```

**Time-Aware Decay Functions** automatically prioritize fresh content by adjusting relevance scores based on document age, with configurable decay rates and function types (exponential, Gaussian, or linear).


### Streamlined Search: Data in, Data Out Experience

The disconnect between raw data and vector embeddings is another pain point for developers using vector databases. Before data reaches Milvus for indexing and vector search, it often undergoes preprocessing using external models that convert raw text, images, or audio into vector representations. After retrieval, additional downstream processing is also required, such as mapping result IDs back to the original content.

Milvus 2.6 simplifies these embedding workflows with the new **Function** interface that integrates third-party embedding models directly into your search pipeline. Instead of pre-computing embeddings, you can now:

1. **Insert raw data directly**: Submit text, images, or other content to Milvus

2. **Configure embedding providers**: Connect to embedding API services from OpenAI, AWS Bedrock, Google Vertex AI, Hugging Face, and more.

3. **Query using natural language**: Search using raw text queries directly

This creates a "Data-In, Data-Out" experience where Milvus streamlines all the behind-the-scenes vector transformations for you.


## Architectural Evolution: Scaling to Tens of Billions of Vectors 

Milvus 2.6 introduces fundamental architectural innovations that enable cost-effective scaling to tens of billions of vectors.


### Replacing Kafka and Pulsar with a New Woodpecker WAL

Previous Milvus deployments relied on external message queues, such as Kafka or Pulsar, as the Write-Ahead Log (WAL) system. While these systems initially worked well, they introduced significant operational complexity and resource overhead.

Milvus 2.6 introduces [**Woodpecker**](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md), a purpose-built, cloud-native WAL system that eliminates these external dependencies through a revolutionary zero-disk design:

- **Everything on object storage**: All log data is persisted in object storage like S3, Google Cloud Storage, or MinIO

- **Distributed metadata**: Metadata is still managed by the etcd key-value store

- **No local disk dependencies**: A choice to eliminate complex architecture and operational overhead involved in distributed local permanent state.

We ran comprehensive benchmarks comparing Woodpecker's performance:

| **System** | **Kafka**   | **Pulsar** | **WP MinIO** | **WP Local** | **WP S3** |
| ---------- | ----------- | ---------- | ------------ | ------------ | --------- |
| Throughput | 129.96 MB/s | 107 MB/s   | 71 MB/s      | 450 MB/s     | 750 MB/s  |
| Latency    | 58 ms       | 35 ms      | 184 ms       | 1.8 ms       | 166 ms    |

Woodpecker consistently reaches 60-80% of the theoretical maximum throughput for each storage backend, with local file system mode achieving 450 MB/s—3.5× faster than Kafka—and S3 mode reaching 750 MB/s, 5.8× higher than Kafka. 

For more details about Woodpecker, check out this blog: [We Replaced Kafka/Pulsar with a Woodpecker for Milvus](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md).


### Search Freshness Achieved Economically

Mission-critical search usually requires newly ingested data to be instantly searchable. Milvus 2.6 replaces message queue dependency to fundamentally improve the handling of fresh updates and provide search freshness at lower resource overhead. The new architecture adds the new **Streaming Node**, a dedicated component that works in close coordination with other Milvus components like the Query Node and Data Node. Streaming Node is built on top of Woodpecker, our lightweight, cloud-native Write-Ahead Log (WAL) system. 

This new component enables: 

- **Great compatibility**: Works with both the new Woodpecker WAL and is backward compatible with Kafka, Pulsar, and other streaming platforms

- **Incremental indexing**: New data becomes searchable immediately, without batch delays

- **Continuous query serving**: Simultaneous high-throughput ingestion and low-latency querying

By isolating streaming from batch processing, the Streaming Node helps Milvus maintain stable performance and search freshness even during high-volume data ingestion. It's designed with horizontal scalability in mind, dynamically scaling node capacity based on data throughput.


### Enhanced Multi-tenancy Capability: Scaling to 100k Collections Per Cluster

Enterprise deployments often require tenant-level isolation. Milvus 2.6 dramatically increases the multi-tenancy support by allowing up to **100,000 collections** per cluster. This is a crucial improvement for organizations running a monolithic large cluster serving many tenants.

This improvement is made possible by numerous engineering optimizations on metadata management, resource allocation, and query planning. Milvus users can now enjoy stable performance even with tens of thousands of collections.


### Other Improvements

Milvus 2.6 offers more architectural enhancements, such as CDC + BulkInsert for simplified data replication across geographic regions and Coord Merge for better cluster coordination in large-scale deployments.


## Getting Started with Milvus 2.6

Milvus 2.6 represents a massive engineering effort with dozens of new features and performance optimizations, developed collaboratively by Zilliz engineers and our amazing community contributors. While we've covered the headline features here, there's more to discover. We highly recommend diving into our comprehensive [release notes](https://milvus.io/docs/release_notes.md) to explore everything this release has to offer! 

Complete documentation, migration guides, and tutorials are available on the[ Milvus website](https://milvus.io/). For questions and community support, join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus).


Introducing Milvus 2.6: Affordable Vector Search at Billion Scale



## Introduction

When building AI-powered applications with [vector databases](https://milvus.io/blog/what-is-a-vector-database.md), understanding system performance becomes critical as your application scales. A single search request might trigger multiple internal operations—vector indexing, similarity calculations, and data retrieval—across different components. Without proper observability, diagnosing slowdowns or failures becomes like finding a needle in a haystack.

**Distributed tracing** solves this problem by tracking requests as they flow through your system, giving you a complete picture of what's happening under the hood.

[**OpenTelemetry (OTEL)**](https://github.com/open-telemetry) is an open-source observability framework backed by the [Cloud Native Computing Foundation (CNCF)](https://www.cncf.io/) that helps you collect traces, metrics, and logs from your applications. It's vendor-neutral, widely adopted, and works seamlessly with popular monitoring tools.

In this guide, we'll show you how to add end-to-end tracing to [**Milvus**](https://milvus.io/), a high-performance vector database built for AI applications. You'll learn to track everything from client requests to internal database operations, making performance optimization and debugging much easier.

We'll also utilize [**Jaeger**](https://github.com/jaegertracing/jaeger-ui) to visualize the trace data, providing you with powerful insights into your vector database operations. 


## What We'll Build

By the end of this tutorial, you'll have a complete tracing pipeline consisting of:

1. **Milvus vector database** with OpenTelemetry tracing enabled

2. **Jaeger** for trace visualization and analysis

3. **A Python client** that automatically traces all Milvus operations

4. **End-to-end visibility** from client requests to database operations

Estimated setup time: 15-20 minutes


## Quick Start (5 Minutes)

Want to see it working first? Here's the fastest path:

1. Clone the demo repository:

```
git clone https://github.com/topikachu/milvus-py-otel
cd milvus-py-otel
```

2. Start the services:

```
docker-compose up -d
```

3. Wait 30 seconds, then check Jaeger UI at: `http://localhost:16686`

4. Run the Python example:

```
pip install -r requirements.txt
python example.py
```

5. Refresh Jaeger and look for traces from both `standalone` (Milvus) and `milvus-client` services.

If you see traces appearing, everything is working! Now let's understand how it all fits together.


## Environment Setup

Here's what you'll need:

- **Milvus 2.5.11** (vector database)

- **Jaeger 1.46.0** (trace visualization)

- **Python 3.7+** (client development)

- **Docker and Docker Compose** (container orchestration)

These versions have been tested together; however, newer versions should also work fine.


## Setting Up Milvus and Jaeger

We'll use Docker Compose to run both services with proper networking and configuration.


### Docker Compose Configuration

Create a `docker-compose.yaml` file:

```
version: '3.7'
Services:
# Milvus - configured to send traces to Jaeger
  milvus:
    image: milvusdb/milvus:v2.5.11
    command: ["milvus", "run", "standalone"]
    environment:
      - ETCD_USE_EMBED=true
      - ETCD_DATA_DIR=/var/lib/milvus/etcd
      - ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml
      - COMMON_STORAGETYPE=local
    volumes:
      - ./embedEtcd.yaml:/milvus/configs/embedEtcd.yaml
      - ./milvus.yaml:/milvus/configs/milvus.yaml
    ports:
      - "19530:19530"
      - "9091:9091"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    security_opt:
      - seccomp:unconfined
    depends_on:
      - jaeger

# Jaeger - starts first since Milvus depends on it
  jaeger:
    image: jaegertracing/all-in-one:1.46.0
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC receiver
      - "4318:4318"    # OTLP HTTP receiver
      - "5778:5778"    # Jaeger agent configs
      - "9411:9411"    # Zipkin compatible endpoint
    environment:
      - COLLECTOR_OTLP_ENABLED=true
```


**Note:** Example configuration files `embedEtcd.yaml` and `milvus.yaml` can be found at: [https://github.com/topikachu/milvus-py-otel](https://github.com/topikachu/milvus-py-otel).


### Milvus Tracing Configuration

Create `configs/milvus.yaml` with the tracing configuration:

```
# OpenTelemetry tracing configuration
trace:
  exporter: otlp           # Use OpenTelemetry Protocol
  sampleFraction: 1.0      # Trace 100% of requests (reduce for production)
  otlp:
    endpoint: jaeger:4317  # Jaeger's OTLP gRPC endpoint
    method: grpc          # Use gRPC protocol
    secure: false         # No TLS (use true in production)
    initTimeoutSeconds: 10
```

Configuration explained:

- `sampleFraction: 1.0` traces every request (useful for development, but use 0.1 or lower in production)

- `secure: false` disables TLS (enable in production)

- `endpoint: jaeger:4317` uses Docker service name for internal communication


### Starting the Services

```
docker-compose up -d
```


### Verifying Trace Delivery from Milvus to Jaeger

Once the services are running, you can verify if trace data is emitted from the Milvus standalone and received by Jaeger. 

- Open your browser and visit Jaeger UI at: `http://localhost:16686/search`

- In the **Search** panel (top-left), choose the **Service** dropdown and select `standalone`. If you see `standalone` in the service list, it means Milvus's built-in OpenTelemetry configuration is working and has successfully pushed trace data to Jaeger.

- Click **Find Traces** to explore trace chains generated by internal Milvus components (such as gRPC interactions between modules).

![](https://assets.zilliz.com/find_traces_811bf9d8ee.png)


### If Trace Data Is Not Showing:

- Double-check that the `trace` block in `milvus.yaml` is configured correctly and that Jaeger is running without issues.

- Inspect the Milvus container logs to see if there are any errors related to Trace initialization.

- Wait a few seconds and refresh the Jaeger UI; trace reporting may experience a short delay.


## Python Client Setup and Dependencies

Now let's set up the Python client to trace all Milvus operations automatically.

First, create a `requirements.txt` file:

```
# OpenTelemetry core
opentelemetry-api==1.33.1
opentelemetry-sdk==1.33.1
# OTLP exporters
opentelemetry-exporter-otlp==1.33.1
opentelemetry-exporter-otlp-proto-grpc==1.33.1
# Automatic gRPC instrumentation
opentelemetry-instrumentation-grpc==0.54b1
# Milvus client
pymilvus==2.5.9
```


Then install dependencies via:

```
pip install -r requirements.txt
```

This ensures your Python environment is ready for tracing gRPC calls made to the Milvus backend.


## Initializing OpenTelemetry in Python

Now, let’s configure tracing within your Python application. This snippet sets up OTEL with gRPC instrumentation and prepares a tracer. 

```
import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.grpc import GrpcInstrumentorClient

# Set OTEL environment variables (you can also load them from external configs)
os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'http://localhost:4317'
os.environ['OTEL_SERVICE_NAME'] = 'milvus-client'

# Define service metadata
resource = Resource.create({
    "service.name": "milvus-client",
    "application": "milvus-otel-test"
})

# Initialize tracer and export processor
trace.set_tracer_provider(
    TracerProvider(resource=resource)
)
otlp_exporter = OTLPSpanExporter()
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# Enable automatic instrumentation for gRPC clients
grpc_client_instrumentor = GrpcInstrumentorClient()
grpc_client_instrumentor.instrument()

# Acquire tracer
tracer = trace.get_tracer(__name__)
```

Here, `GrpcInstrumentorClient()` hooks into the underlying gRPC stack so you don’t need to manually modify client code for instrumentation. The `OTLPSpanExporter()` is configured to send trace data to your local Jaeger instance.


## Complete Milvus Python Example with Tracing

Now let's create a comprehensive example that demonstrates tracing with realistic Milvus operations:

```
from pymilvus import MilvusClient
from opentelemetry import trace

with tracer.start_as_current_span("test_milvus_otel"):
    milvus_client = MilvusClient(
        uri="http://localhost:19530",
    )
    collection_name = "quick_setup"

    # Drop collection if it exists
    if milvus_client.has_collection(collection_name):
        milvus_client.drop_collection(collection_name)

    # Create collection
    milvus_client.create_collection(
        collection_name=collection_name,
        dimension=5
    )

    # Add additional operations here
    
    milvus_client.close()
```


## Viewing Trace Output

Once your Python client sends trace data, return to Jaeger: [`http://localhost:16686`](http://localhost:16686)

Select the `milvus-client` service to view trace spans that correspond to your Python client’s Milvus operations. This makes it much easier to analyze performance and trace interactions across system boundaries.

![](https://assets.zilliz.com/milvus_client_22aab6ab9f.png)


## Examples in Other Languages

Beyond Python, you can implement Milvus tracing in other languages:

👉[**Java**](https://github.com/topikachu/milvus-java-otel): Use the OpenTelemetry Java Agent for zero-code instrumentation
👉[**Go**](https://github.com/topikachu/milvus-go-otel): Leverage the OpenTelemetry Go SDK for native integration
👉[**Node.js**](https://github.com/topikachu/milvus-nodejs-otel): Auto-instrument gRPC calls with the JavaScript SDK

Each example follows similar patterns but uses language-specific OpenTelemetry libraries.


## Summary

You've successfully implemented end-to-end tracing for Milvus operations! Here's what you've accomplished:

- ✅ **Infrastructure**: Set up Milvus and Jaeger with proper networking

- ✅ **Server-side tracing**: Configured Milvus to export traces automatically

- ✅ **Client-side tracing**: Instrumented Python client with OpenTelemetry

- ✅ **Visualization**: Used Jaeger to analyze system performance

- ✅ **Production readiness**: Learned configuration best practices

All works without any changes to the Milvus SDK source code. Just a few configuration settings and your tracing pipeline is live—simple, effective, and ready for production.

You can take it further by integrating logs and metrics to build a complete monitoring solution for your AI-native vector database deployment.


## Learn More 

- Milvus Documentation: [https://milvus.io/docs](https://milvus.io/docs)

- OpenTelemetry for Python: [https://opentelemetry.io/docs/instrumentation/python/](https://opentelemetry.io/docs/instrumentation/python/)

- Jaeger Documentation: [https://www.jaegertracing.io/docs/](https://www.jaegertracing.io/docs/)

- Milvus OpenTelemetry Integration Demo (Python): [https://github.com/topikachu/milvus-py-otel](https://github.com/topikachu/milvus-py-otel)

How to Set Up OpenTelemetry with Milvus for End-to-End Request Tracing


It has never been easier to build Generative AI applications. A rich ecosystem of tools, AI models, and datasets allows even non-specialized software engineers to build impressive chatbots, image generators, and more. This tooling, for the most part, is made for Python and builds on top of PyTorch. But what about when you don’t have access to Python in production and need to use Java, Golang, Rust, C++, or another language?

We will restrict ourselves to model inference, including both embedding models and foundation models; other tasks, such as model training and fine-tuning, are not typically completed at deployment time. What are our options for model inference without Python? The most obvious solution is to utilize an online service from providers like Anthropic or Mistral. They typically provide an SDK for languages other than Python, and if they didn’t, it would require only simple REST API calls. But what if our solution has to be entirely local due to, for example, compliance or privacy concerns?

Another solution is to run a Python server locally. The original problem was posed as being unable to run Python in production, so that rules out using a local Python server. Related local solutions will likely suffer similar legal, security-based, or technical restrictions. _We need a fully contained solution that allows us to call the model directly from Java or another non-Python language._

![](https://assets.zilliz.com/A_Python_metamorphoses_into_an_Onyx_butterfly_a65c340c47.png)

_Figure 1: A Python metamorphoses into an Onyx butterfly._


## What is ONNX (Open Neural Network Exchange)? 
[ONNX](https://github.com/onnx/onnx) (Open Neural Network Exchange) is a platform-agnostic ecosystem of tools for performing neural network model inference. It was initially developed by the PyTorch team at Meta (then Facebook), with further contributions from Microsoft, IBM, Huawei, Intel, AMD, Arm, and Qualcomm. Currently, it is an open-source project owned by the Linux Foundation for AI and Data. ONNX is the de facto method for distributing platform-agnostic neural network models.

![](https://assets.zilliz.com/A_partial_ONNX_computational_graph_for_a_NN_transformer_11deebefe0.png)

_Figure 2: A (partial) ONNX computational graph for a NN transformer_

**We typically use “ONNX” in a narrower sense to refer to its file format.** An ONNX model file represents a computational graph, often including the weight values of a mathematical function, and the standard defines common operations for neural networks. You can think of it similarly to the computational graph created when you use autodiff with PyTorch. From another perspective, the ONNX file format serves as an _intermediate representation_ (IR) for neural networks, much like native code compilation, which also involves an IR step. See the illustration above visualizing an ONNX computational graph.

![](https://assets.zilliz.com/An_IR_allows_many_combinations_of_front_ends_and_back_ends_a05e259849.png)

_Figure 3: An IR allows many combinations of front-ends and back-ends_

The ONNX file format is just one part of the ONNX ecosystem, which also includes libraries for manipulating computational graphs and libraries for loading and running ONNX model files. These libraries span languages and platforms. Since ONNX is just an IR (Intermediate Representation Language), optimizations specific to a given hardware platform can be applied before running it with native code. See the figure above illustrating combinations of front-ends and back-ends.


## ONNX Workflow
For discussion purposes, we will investigate calling a text embedding model from Java, for example, in preparation for data ingestion to the open-source vector database [Milvus](https://milvus.io/). So, if we are to call our embedding or foundation model from Java, is it as simple as using the ONNX library on the corresponding model file? Yes, but we will need to procure files for both the model and the tokenizer encoder (and decoder for foundation models). We can produce these ourselves using Python offline, that is, before production, which we now explain.


## Exporting NN Models from Python
Let’s open a common text embedding model, `all-MiniLM-L6-v2`, from Python using HuggingFace’s sentence-transformers library. We will use the HF library indirectly via .txtai’s util library since we need a wrapper around sentence-transformers that also exports the pooling and normalization layers after the transformer function. (These layers take the context-dependent token embeddings, that is, the output of the transformer, and transform it into a single text embedding.)

```
from txtai.pipeline import HFOnnx

path = "sentence-transformers/all-MiniLM-L6-v2"
onnx_model = HFOnnx()
model = onnx_model(path, "pooling", "model.onnx", True)
```

We instruct the library to export `sentence-transformers/all-MiniLM-L6-v2` from the HuggingFace model hub as ONNX, specifying the task as text embedding and enabling model quantization. Calling `onnx_model()` will download the model from the model hub if it does not already exist locally, convert the three layers to ONNX, and combine their computational graphs.

Are we ready now to perform inference in Java? Not quite so fast. The model inputs a list of tokens (or a list of lists for more than one sample) corresponding to the tokenization of the text we wish to embed. Therefore, unless we can perform all tokenization before production time, we will need to run the tokenizer from within Java.

There are a few options for this. One involves either implementing or finding an implementation of the tokenizer for the model in question in Java or another language, and calling it from Java as a static or dynamically linked library. An easier solution is to convert the tokenizer to an ONNX file and use it from Java, just as we use the model ONNX file.

Plain ONNX, however, does not contain the necessary operations to implement the computational graph of a tokenizer. For this reason, Microsoft created a library to augment ONNX called ONNXRuntime-Extensions. It defines useful operations for all manner of data pre- and postprocessing, not only text tokenizers.

Here is how we export our tokenizer as an ONNX file:

```
from onnxruntime_extensions import gen_processing_models
from sentence_transformers import SentenceTransformer

embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
tok_encode, _ = gen_processing_models(embedding_model.tokenizer, pre_kwargs={})

onnx_tokenizer_path = "tokenizer.onnx"
with open(tokenizer_path, "wb") as f:
  f.write(tok_encode.SerializeToString())
```

We have discarded the decoder of the tokenizer, since embedding sentences doesn’t require it. Now, we have two files: `tokenizer.onnx` for tokenizing text, and `model.onnx` for embedding strings of tokens.


## Model Inference in Java
Running our model from within Java is now trivial. Here are some of the important lines of code from the full example:

```java
// Imports required for Java/ONNX integration
import ai.onnxruntime.*;
import ai.onnxruntime.extensions.*;

…

// Set up inference sessions for tokenizer and model
var env = OrtEnvironment.getEnvironment();

var sess_opt = new OrtSession.SessionOptions();
sess_opt.registerCustomOpLibrary(OrtxPackage.getLibraryPath());

var tokenizer = env.createSession("app/tokenizer.onnx", sess_opt);
var model = env.createSession("app/model.onnx", sess_opt);

…

// Perform inference and extract text embeddings into native Java
var results = session.run(inputs).get("embeddings");
float[][] embeddings = (float[][]) results.get().getValue();
```

A full working example can be found in the resources section.


## Summary 
We have seen in this post how it is possible to export open-source models from HuggingFace’s model hub and use them directly from languages other than Python. We note, however, some caveats:

First, the ONNX libraries and runtime extensions have varying levels of feature support. It may not be possible to use all models across all languages until a future SDK update is released. The ONNX runtime libraries for Python, C++, Java, and JavaScript are the most comprehensive. 

Second, the HuggingFace hub contains pre-exported ONNX, but these models don’t include the final pooling and normalization layers. You should be aware of how `sentence-transformers` works if you intend to use `torch.onnx` directly.

Nevertheless, ONNX has the backing of major industry leaders and is on a trajectory to become a frictionless means of cross-platform Generative AI.


## Resources
- [Example onnx code in Python and Java](https://github.com/milvus-io/bootcamp/tree/master/tutorials/quickstart/onnx_example)

- <https://onnx.ai/>

- <https://onnxruntime.ai/>

- <https://onnxruntime.ai/docs/extensions/>

- <https://milvus.io/blog> 

- <https://github.com/milvus-io/bootcamp/tree/master> 


No Python, No Problem: Model Inference with ONNX in Java, or Any Other Language





For decades, `SELECT * FROM WHERE` has been the golden rule of database queries. Whether for reporting systems, financial analysis, or user behavior queries, we've grown accustomed to using structured language to precisely manipulate data. Even NoSQL, which once proclaimed an "anti-SQL revolution," eventually caved and introduced SQL support, acknowledging its seemingly irreplaceable position.

_But have you ever wondered: we've spent over 50 years teaching computers to speak human language, so why are we still forcing humans to speak "computer"?_

**Whether you like it or not, here's the truth: SQL is destined for decline in the era of AI.** It may still be used in legacy systems, but it's becoming increasingly irrelevant for modern AI applications. The AI revolution isn't just changing how we build software—it's making SQL obsolete, and most developers are too busy optimizing their JOINs to notice. 


![](https://assets.zilliz.com/why_ai_databases_don_t_need_SQL_2d12f615df.png)


## Natural Language: The New Interface for AI Databases 

The future of database interaction isn't about learning better SQL—it's about **abandoning syntax entirely**. 

Instead of wrestling with complex SQL queries, imagine simply saying:

_"Help me find users whose recent purchasing behavior is most similar to our top customers from last quarter."_

The system understands your intent and automatically decides:

- Should it query structured tables or perform a vector similarity search across user embeddings?

- Should it call external APIs to enrich the data?

- How should it rank and filter the results?

All completed automatically. No syntax. No debugging. No Stack Overflow searches for "how to do a window function with multiple CTEs." You're no longer a database "programmer"—you're having a conversation with an intelligent data system.

This isn't science fiction. According to Gartner predictions, by 2026, most enterprises will prioritize natural language as their primary query interface, with SQL changing from a "must-have" to an "optional" skill.

The transformation is already happening:

**✅ Zero syntax barriers:** Field names, table relationships, and query optimization become the system's problem, not yours

**✅ Unstructured data friendly:** Images, audio, and text become first-class query objects

**✅ Democratized access:** Operations teams, product managers, and analysts can directly query data as easily as your senior engineer


## Natural Language Is Just the Surface; AI Agents Are the Real Brain

Natural language queries are just the tip of the iceberg. The real breakthrough is [AI agents](https://zilliz.com/blog/what-exactly-are-ai-agents-why-openai-and-langchain-are-fighting-over-their-definition) that can reason about data like humans do.

Understanding human speech is step one. Understanding what you want and executing it efficiently—that's where the magic happens.

AI agents serve as the database's "brain," handling:

- **🤔 Intent understanding:** Determining which fields, databases, and indexes you actually need

- **⚙️ Strategy selection:** Choosing between structured filtering, vector similarity, or hybrid approaches

- **📦 Capability orchestration:** Executing APIs, triggering services, coordinating cross-system queries

- **🧾 Intelligent formatting:** Returning results you can immediately understand and act on

Here's what this looks like in practice. In the [Milvus vector database,](https://milvus.io/) a complex similarity search becomes trivial:

```
results = collection.search(query_vector, top_k=10, filter="is_active == true")
```


**One line. No JOINs. No subqueries. No performance tuning.** The [vector database](https://zilliz.com/learn/what-is-vector-database) handles semantic similarity while traditional filters handle exact matches. It's faster, simpler, and actually understands what you want.

This "API-first" approach naturally integrates with large language models' [Function Calling](https://zilliz.com/blog/function-calling-vs-mcp-vs-a2a-developers-guide-to-ai-agent-protocols) capabilities—faster execution, fewer errors, easier integration.


## Why SQL Falls Apart in the AI Era

SQL was designed for a structured world. However, the AI-driven future will be dominated by unstructured data, semantic understanding, and intelligent retrieval—everything SQL was never built to handle.

Modern applications are inundated with unstructured data, including text embeddings from language models, image vectors from computer vision systems, audio fingerprints from speech recognition, and multimodal representations that combine text, images, and metadata.

This data doesn't fit neatly into rows and columns—it exists as vector embeddings in high-dimensional semantic space, and SQL has absolutely no idea what to do with it.


### SQL + Vector: A Beautiful Idea That Executes Poorly

Desperate to stay relevant, traditional databases are bolting vector capabilities onto SQL. PostgreSQL added the `<->` operator for vector similarity search:

```
SELECT *
  FROM items
 ORDER BY embedding <-> query_vector
 LIMIT 10;
```


This looks clever, but it's fundamentally flawed. You're forcing vector operations through SQL parsers, query optimizers, and transaction systems designed for a completely different data model.

The performance penalty is brutal:

📊 **Real benchmark data**: Under identical conditions, purpose-built Milvus delivers 60% lower query latency and 4.5x higher throughput compared to PostgreSQL with pgvector.

Why such poor performance? Traditional databases create unnecessarily complex execution paths:

- **Parser overhead**: Vector queries get forced through SQL syntax validation

- **Optimizer confusion**: Query planners optimized for relational joins struggle with similarity searches

- **Storage inefficiency**: Vectors stored as BLOBs require constant encoding/decoding

- **Index mismatch**: B-trees and LSM structures are completely wrong for high-dimensional similarity search


### Relational vs AI/Vector Databases: Fundamentally Different Philosophies

The incompatibility runs deeper than performance. These are entirely different approaches to data:

| **Aspect**        | **SQL/Relational Databases**                             | **Vector/AI Databases**                                                            |
| ----------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Data Model        | Structured fields (numbers, strings) in rows and columns | High-dimensional vector representations of unstructured data (text, images, audio) |
| Query Logic       | Exact matching + boolean operations                      | Similarity matching + semantic search                                              |
| Interface         | SQL                                                      | Natural language + Python APIs                                                     |
| Philosophy        | ACID compliance, perfect consistency                     | Optimized recall, semantic relevance, real-time performance                        |
| Index Strategy    | B+ trees, hash indexes etc.                              | HNSW, IVF, product quantization etc.                                               |
| Primary Use Cases | Transactions, reporting, analytics                       | Semantic search, multimodal search, recommendations, RAG systems, AI agents        |

Trying to make SQL work for vector operations is like using a screwdriver as a hammer—not technically impossible, but you're using the wrong tool for the job.


## Vector Databases: Purpose-Built for AI 

Vector databases like [Milvus](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/) aren't "SQL databases with vector features"—they're intelligent data systems designed from the ground up for AI-native applications.


### 1. Native Multimodal Support

Real AI applications don't just store text—they work with images, audio, video, and complex nested documents. Vector databases handle diverse data types and multi-vector structures like [ColBERT](https://zilliz.com/learn/explore-colbert-token-level-embedding-and-ranking-model-for-similarity-search) and [ColPALI](https://zilliz.com/blog/colpali-milvus-redefine-document-retrieval-with-vision-language-models), adapting to rich semantic representations from different AI models.


### 2. Agent-Friendly Architecture

Large language models excel at function calling, not SQL generation. Vector databases offer Python-first APIs that integrate seamlessly with AI agents, enabling the completion of complex operations, such as vector retrieval, filtering, reranking, and semantic highlighting, all within a single function call, without requiring a query language translation layer.


### 3. Semantic Intelligence Built-In

Vector databases don't just execute commands—**they understand intent.** Working with AI agents and other AI applications, they break free from literal keyword matching to achieve true semantic retrieval. They know not just "how to query" but "what you really want to find."


### 4. Optimized for Relevance, Not Just Speed

Like large language models, vector databases strike a balance between performance and recall. Through metadata filtering, [hybrid vector and full-text search](https://milvus.io/blog/get-started-with-hybrid-semantic-full-text-search-with-milvus-2-5.md), and reranking algorithms, they continuously improve result quality and relevancy, finding content that's actually valuable, not just fast to retrieve.


## The Future of Databases is Conversational

Vector databases represent a fundamental shift in how we think about data interaction. They're not replacing relational databases—they're purpose-built for AI workloads and addressing entirely different problems in an AI-first world.

Just as large language models didn't upgrade traditional rule engines but redefined human-machine interaction entirely, vector databases are redefining how we find and work with information.

We're transitioning from "languages written for machines to read" to "systems that understand human intent." Databases are evolving from rigid query executors to intelligent data agents that understand context and proactively surface insights.

The developers building AI applications today don't want to write SQL—they want to describe what they need and let intelligent systems figure out how to get it.

So next time you need to find something in your data, try a different approach. Don't write a query—just say what you're looking for. Your database might surprise you by actually understanding what you mean.

_And if it doesn't? Maybe it's time to upgrade your database, not your SQL skills._


Why AI Databases Don't Need SQL



**Probably every AI developer has built a RAG system that works perfectly... in their local environment.**

You've nailed the retrieval accuracy, optimized your vector database, and your demo runs like butter. Then you deploy to production and suddenly:

- Your 200ms local queries take 3 seconds for actual users

- Colleagues in different regions report completely different performance

- The embedding provider you chose for "best accuracy" becomes your biggest bottleneck

What happened? Here's the performance killer no one benchmarks: **embedding API latency**.

While MTEB rankings obsess over recall scores and model sizes, they ignore the metric your users feel—how long they wait before seeing any response. We tested every major embedding provider across real-world conditions and discovered extreme latency differences they'll make you question your entire provider selection strategy.

**_Spoiler: The most popular embedding APIs aren't the fastest. Geography matters more than model architecture. And sometimes a $20/month CPU beats a $200/month API call._**


## Why Embedding API Latency Is the Hidden Bottleneck in RAG

When building RAG systems, e-commerce search, or recommendation engines, embedding models are the core component that transforms text into vectors, enabling machines to understand semantics and perform efficient similarity searches. While we typically pre-compute embeddings for document libraries, user queries still require real-time embedding API calls to convert questions into vectors before retrieval, and this real-time latency often becomes the performance bottleneck in the entire application chain.

Popular embedding benchmarks like MTEB focus on recall accuracy or model size, often overlooking the crucial performance metric—API latency. Using Milvus's `TextEmbedding` Function, we conducted comprehensive real-world tests on major embedding service providers in North America and Asia. 

Embedding latency manifests at two critical stages:


### Query-Time Impact

In a typical RAG workflow, when a user asks a question, the system must:

- Convert the query to a vector via an embedding API call

- Search for similar vectors in Milvus

- Feed results and the original question to an LLM

- Generate and return the answer

Many developers assume the LLM's answer generation is the slowest part. However, many LLMs’ streaming output capability creates an illusion of speed—you see the first token quickly. In reality, if your embedding API call takes hundreds of milliseconds or even seconds, it becomes the first—and most noticeable—bottleneck in your response chain.


### Data Ingestion Impact

Whether building an index from scratch or performing routine updates, bulk ingestion requires vectorizing thousands or millions of text chunks. If each embedding call experiences high latency, your entire data pipeline slows dramatically, delaying product releases and knowledge base updates.

Both situations make embedding API latency a non-negotiable performance metric for production RAG systems.


## Measuring Real-World Embedding API Latency with Milvus

Milvus is an open-source, high-performance vector database that offers a new `TextEmbedding` Function interface. This feature directly integrates popular embedding models from OpenAI, Cohere, AWS Bedrock, Google Vertex AI, Voyage AI, and many more providers into your data pipeline, streamlining your vector search pipeline with a single call.

Using this new function interface, we tested and benchmarked popular embedding APIs from well-known providers like OpenAI and Cohere, as well as others like AliCloud and SiliconFlow, measuring their end-to-end latency in realistic deployment scenarios.

Our comprehensive test suite covered various model configurations:


| **Provider**     | **Model**                             | **Dimensions** |                  
| ---------------- | ------------------------------------- | -------------- |
| OpenAI           | text-embedding-ada-002                | 1536           |
| OpenAI           | text-embedding-3-small                | 1536           |
| OpenAI           | text-embedding-3-large                | 3072           |
| AWS Bedrock      | amazon.titan-embed-text-v2:0          | 1024           |
| Google Vertex AI | text-embedding-005                    | 768            |
| Google Vertex AI | text-multilingual-embedding-002       | 768            |
| VoyageAI         | voyage-3-large                        | 1024           |
| VoyageAI         | voyage-3                              | 1024           |
| VoyageAI         | voyage-3-lite                         | 512            |
| VoyageAI         | voyage-code-3                         | 1024           |
| Cohere           | embed-english-v3.0                    | 1024           |
| Cohere           | embed-multilingual-v3.0               | 1024           |
| Cohere           | embed-english-light-v3.0              | 384            |
| Cohere           | embed-multilingual-light-v3.0         | 384            |
| Aliyun Dashscope | text-embedding-v1                     | 1536           |
| Aliyun Dashscope | text-embedding-v2                     | 1536           |
| Aliyun Dashscope | text-embedding-v3                     | 1024           |
| Siliconflow      | BAAI/bge-large-zh-v1.5                | 1024           |
| Siliconflow      | BAAI/bge-large-en-v1.5                | 1024           |
| Siliconflow      | netease-youdao/bce-embedding-base_v1  | 768            |
| Siliconflow      | BAAI/bge-m3                           | 1024           |
| Siliconflow      | Pro/BAAI/bge-m3                       | 1024           |
| TEI              | BAAI/bge-base-en-v1.5                 | 768            |


## 7 Key Findings from Our Benchmarking Results 

We tested leading embedding models under varying batch sizes, token lengths, and network conditions, measuring median latency across all scenarios. The results uncover key insights that could reshape how you choose and optimize embedding APIs. Let’s take a look.


### 1. Global Network Effects Are More Significant Than You Think

The network environment is perhaps the most critical factor affecting embedding API performance. The same embedding API service provider can perform dramatically differently across network environments. 

![](https://assets.zilliz.com/latency_in_Asia_vs_in_US_cb4b5a425a.png)

When your application is deployed in Asia and accesses services like OpenAI, Cohere, or VoyageAI deployed in North America, network latency increases significantly. Our real-world tests show API call latency universally increased by **3 to 4 times**!

Conversely, when your application is deployed in North America and accesses Asian services like AliCloud Dashscope or SiliconFlow, performance degradation is even more severe. SiliconFlow, in particular, showed latency increases of **nearly 100 times** in cross-region scenarios!

This means you must always select embedding providers based on your deployment location and user geography—performance claims without network context are meaningless.


### 2. Model Performance Rankings Reveal Surprising Results

Our comprehensive latency testing revealed clear performance hierarchies:

- **North America-based Models (median latency)**: Cohere > Google Vertex AI > VoyageAI > OpenAI > AWS Bedrock

- **Asia-based Models (median latency)**: SiliconFlow > AliCloud Dashscope

These rankings challenge conventional wisdom about provider selection. 

![](https://assets.zilliz.com/median_latency_with_batch_size_1_ef83bec9c8.png)

![](https://assets.zilliz.com/median_latency_with_batch_size_10_0d4e52566f.png)

![](https://assets.zilliz.com/all_model_latency_vs_token_length_when_batch_size_is_10_537516cc1c.png)

![](https://assets.zilliz.com/all_model_latency_vstoken_lengthwhen_batch_size_is_10_4dcf0d549a.png)

Note: Due to the significant impact of network environment and server geographic regions on real-time embedding API latency, we compared North America and Asia-based model latencies separately.


### 3. Model Size Impact Varies Dramatically by Provider

We observed a general trend where larger models have higher latency than standard models, which have higher latency than smaller/lite models. However, this pattern wasn't universal and revealed important insights about backend architecture. For example: 

- **Cohere and OpenAI** showed minimal performance gaps between model sizes

- **VoyageAI** exhibited clear performance differences based on model size

![](https://assets.zilliz.com/Model_Size_Impact_Varies_Dramatically_by_Provider_1_f9eaf2be26.png)


![](https://assets.zilliz.com/Model_Size_Impact_Varies_Dramatically_by_Provider_2_cf4d72d1ad.png)


![](https://assets.zilliz.com/Model_Size_Impact_Varies_Dramatically_by_Provider_3_5e0c8d890b.png)


This indicates that API response time depends on multiple factors beyond model architecture, including backend batching strategies, request handling optimization, and provider-specific infrastructure. The lesson is clear: _don't trust model size or release date as reliable performance indicators—always test in your own deployment environment._


### 4. Token Length and Batch Size Create Complex Trade-offs

Depending on your backend implementation, especially your batching strategy. Token length may have little impact on latency until batch sizes grow. Our testing revealed some clear patterns:

- **OpenAI's latency** remained fairly consistent between small and large batches, suggesting generous backend batching capabilities

- **VoyageAI** showed clear token-length effects, implying minimal backend batching optimization

Larger batch sizes increase absolute latency but improve overall throughput. In our tests, moving from batch=1 to batch=10 increased latency by 2×-5×, while substantially boosting total throughput. This represents a critical optimization opportunity for bulk processing workflows where you can trade individual request latency for dramatically improved overall system throughput.

![](https://assets.zilliz.com/Going_from_batch_1_to_10_latency_increased_2_5_9811536a3c.png)

Going from batch=1 to 10, latency increased 2×–5×


### 5. API Reliability Introduces Production Risk

We observed significant variability in latency, particularly with OpenAI and VoyageAI, introducing unpredictability into production systems. 

![](https://assets.zilliz.com/Latency_variance_when_batch_1_d9cd88fb73.png)

Latency variance when batch=1

![](https://assets.zilliz.com/Latency_variance_when_batch_10_5efc33bf4e.png)

Latency variance when batch=10

While our testing focused primarily on latency, relying on any external API introduces inherent failure risks, including network fluctuations, provider rate limiting, and service outages. Without clear SLAs from providers, developers should implement robust error handling strategies, including retries, timeouts, and circuit breakers to maintain system reliability in production environments.


### 6. Local Inference Can Be Surprisingly Competitive

Our tests also revealed that deploying mid-sized embedding models locally can offer performance comparable to cloud APIs—a crucial finding for budget-conscious or latency-sensitive applications.

For example, deploying the open-source `bge-base-en-v1.5` via TEI (Text Embeddings Inference) on a modest 4c8g CPU matched SiliconFlow's latency performance, providing an affordable local-inference alternative. This finding is particularly significant for individual developers and small teams who may lack enterprise-grade GPU resources but still need high-performance embedding capabilities.

![](https://assets.zilliz.com/TEI_Latency_2f09be1ef0.png)

TEI Latency

### 7. Milvus Overhead Is Negligible

Since we used Milvus to test embedding API latency, we validated that the additional overhead introduced by Milvus's TextEmbedding Function is minimal and virtually negligible. Our measurements show Milvus operations add only 20-40ms in total while embedding API calls take hundreds of milliseconds to several seconds, meaning Milvus adds less than 5% overhead to the total operation time. The performance bottleneck primarily lies in network transmission and the embedding API service providers' processing capabilities, not in the Milvus server layer.

## Tips: How to Optimize Your RAG Embedding Performance

Based on our benchmarks, we recommend the following strategies to optimize your RAG system's embedding performance:


### 1. Always Localize Your Testing

Don't trust any generic benchmark reports (including this one!). You should always test models within your actual deployment environment rather than relying solely on published benchmarks. Network conditions, geographic proximity, and infrastructure differences can dramatically impact real-world performance.


### 2. Geo-Match Your Providers Strategically

- **For North American deployments**: Consider Cohere, VoyageAI, OpenAI/Azure, or GCP Vertex AI—and always conduct your own performance validation

- **For Asian deployments**: Seriously consider Asian model providers such as AliCloud Dashscope or SiliconFlow, which offer better regional performance

- **For global audiences**: Implement multi-region routing or select providers with globally distributed infrastructure to minimize cross-region latency penalties


### 3. Question Default Provider Choices

OpenAI's embedding models are so popular that many enterprises and developers choose them as default options. However, our tests revealed that OpenAI's latency and stability were average at best, despite its market popularity. Challenge assumptions about "best" providers with your own rigorous benchmarks—popularity doesn't always correlate with optimal performance for your specific use case.


### 4. Optimize Batch and Chunk Configurations

One configuration doesn't fit all models or use cases. The optimal batch size and chunk length vary significantly between providers due to different backend architectures and batching strategies. Experiment systematically with different configurations to find your optimal performance point, considering the throughput vs. latency trade-offs for your specific application requirements.


### 5. Implement Strategic Caching

For high-frequency queries, cache both the query text and its generated embeddings (using solutions like Redis). Subsequent identical queries can directly hit the cache, reducing latency to milliseconds. This represents one of the most cost-effective and impactful query latency optimization techniques. 


### 6. Consider Local Inference Deployment

If you have extremely high requirements for data ingestion latency, query latency, and data privacy, or if API call costs are prohibitive, consider deploying embedding models locally for inference. Standard API plans often come with QPS limitations, unstable latency, and lack of SLA guarantees—constraints that can be problematic for production environments.


For many individual developers or small teams, the lack of enterprise-grade GPUs is a barrier to local deployment of high-performance embedding models. However, this doesn't mean abandoning local inference entirely. With high-performance inference engines like [Hugging Face's text-embeddings-inference](https://github.com/huggingface/text-embeddings-inference), even running small to medium-sized embedding models on a CPU can achieve decent performance that may outperform high-latency API calls, especially for large-scale offline embedding generation. 


This approach requires careful consideration of trade-offs between cost, performance, and maintenance complexity.


## How Milvus Simplifies Your Embedding Workflow

As mentioned, Milvus is not just a high-performance vector database—it also offers a convenient embedding function interface that seamlessly integrates with popular embedding models from various providers such as OpenAI, Cohere, AWS Bedrock, Google Vertex AI, Voyage AI, and more across the world into your vector search pipeline.

Milvus goes beyond vector storage and retrieval with features that streamline embedding integration:

- **Efficient Vector Management**: As a high-performance database built for massive vector collections, Milvus offers reliable storage, flexible indexing options (HNSW, IVF, RaBitQ, DiskANN, and more), and fast, accurate retrieval capabilities.

- **Streamlined Provider Switching**: Milvus offers a `TextEmbedding` Function interface, allowing you to configure the function with your API keys, switch providers or models instantly, and measure real-world performance without complex SDK integration.

- **End-to-End Data Pipelines**: Call `insert()` with raw text, and Milvus automatically embeds and stores vectors in one operation, dramatically simplifying your data pipeline code.

- **Text-to-Results in One Call**: Call `search()` with text queries, and Milvus handles embedding, searching, and returning results—all in a single API call.

- **Provider-Agnostic Integration**: Milvus abstracts provider implementation details; just configure your Function and API key once, and you're ready to go.

- **Open-Source Ecosystem Compatibility**: Whether you generate embeddings via our built-in `TextEmbedding` Function, local inference, or another method, Milvus provides unified storage and retrieval capabilities.

This creates a streamlined "Data-In, Insight-Out" experience where Milvus handles vector generation internally, making your application code more straightforward and maintainable.


## Conclusion: The Performance Truth Your RAG System Needs

The silent killer of RAG performance isn't where most developers are looking. While teams pour resources into prompt engineering and LLM optimization, embedding API latency quietly sabotages user experience with delays that can be 100x worse than expected. Our comprehensive benchmarks expose the harsh reality: popular doesn't mean performant, geography matters more than algorithm choice in many cases, and local inference sometimes beats expensive cloud APIs.

These findings highlight a crucial blind spot in RAG optimization. Cross-region latency penalties, unexpected provider performance rankings, and the surprising competitiveness of local inference aren't edge cases—they're production realities affecting real applications. Understanding and measuring embedding API performance is essential for delivering responsive user experiences.

Your embedding provider choice is one critical piece of your RAG performance puzzle. By testing in your actual deployment environment, selecting geographically appropriate providers, and considering alternatives like local inference, you can eliminate a major source of user-facing delays and build truly responsive AI applications.

For more details on how we did this benchmarking, check [this notebook](https://github.com/zhuwenxing/text-embedding-bench).


We Benchmarked 20+ Embedding APIs with Milvus: 7 Insights That Will Surprise You



The key operation of [vector databases](https://milvus.io/blog/what-is-a-vector-database.md) is _similarity search_, which involves finding the nearest neighbors in the database to a query vector, for example, by Euclidean distance. A naive method would calculate the distance from the query vector to every vector stored in the database and take the top-K closest. However, this clearly does not scale as the size of the database grows. In practice, a naive similarity search is practical only for databases with fewer than around 1 million vectors. How are we to scale our search to the 10s and 100s of millions, or even to the billions of vectors?

![](https://assets.zilliz.com/Figure_Descending_a_hierarchy_of_vector_search_indices_cf9fb8060a.png)

_Figure: Descending a hierarchy of vector search indices_

Many algorithms and data structures have been developed to scale similarity search in high-dimensional vector spaces to sub-linear time complexity. In this article, we’ll explain and implement a popular and effective method called Hierarchical Navigable Small Worlds (HNSW), which is frequently the default choice for medium-sized vector datasets. It belongs to the family of search methods that construct a graph over the vectors, where vertices denote vectors and edges denote similarity between them. Search is performed by navigating the graph, in the simplest case, greedily traversing to the neighbor of the current node that is closest to the query and repeating until a local minimum is reached.

We will explain in more detail how the search graph is constructed, how the graph enables search, and at the end, link to an HNSW implementation, by yours truly, in simple Python.


## Navigable Small Worlds

![](https://assets.zilliz.com/Figure_NSW_graph_created_from_100_randomly_located_2_D_points_3ffccbd6a7.jpg)

_Figure: NSW graph created from 100 randomly located 2D points._

As mentioned, HNSW constructs a search graph offline before we can perform a query. The algorithm builds on top of prior work, a method called Navigable Small Worlds (NSW). We will explain NSW first and it will then be trivial to go from there to _Hierarchical_ NSW. The illustration above is of a constructed search graph for NSW over 2-dimensional vectors. In all examples below, we restrict ourselves to 2-dimensional vectors so as to be able to visualize them.


## Constructing the Graph

An NSW is a graph where the vertices represent vectors and the edges are constructed heuristically from the similarity between vectors so that most vectors are reachable from anywhere via a small number of hops. This is the so-called “small world” property that permits quick navigation. See the above figure.

The graph is initialized to be empty. We iterate through the vectors, adding each to the graph in turn. For each vector, starting at a random entry node, we greedily find the closest R nodes reachable from the entry point _in the graph so far constructed_. These R nodes are then connected to a new node representing the vector being inserted, optionally pruning any neighboring nodes that now have more than R neighbors. Repeating this process for all vectors will result in the NSW graph. See the above illustration visualizing the algorithm, and refer to the resources at the end of the article for a theoretical analysis of the properties of a graph constructed like this.


## Searching the Graph

We have already seen the search algorithm from its use in graph construction. In this case, however, the query node is provided by the user, rather than being one for insertion into the graph. Starting from a random entry note, we greedily navigate to its neighbor closest to the query, maintaining a dynamic set of the closest vectors encountered so far. See the illustration above. Note that we can enhance search accuracy by initiating searches from multiple random entry points and aggregating the results, along with considering multiple neighbors at each step. However, these improvements come at the cost of increased latency.


# Adding Hierarchy

![](https://assets.zilliz.com/adding_hierarchy_0101234812.png)

So far, we have described the NSW algorithm and data structure that can help us scale up search in high-dimensional space. Nonetheless, the method suffers serious shortcomings, including failure in low dimensions, slow search convergence, and a tendency to be trapped in local minima.

The authors of HNSW fix these shortcomings with three modifications to NSW:

- Explicit selection of the entry nodes during construction and search;

- Separation of edges by different scales; and, 

- Use of an advanced heuristic to select the neighbors.

The first two are realized with a simple idea: building _a hierarchy of search graphs_. Instead of a single graph, as in NSW, HNSW constructs a hierarchy of graphs. Each graph, or layer, is individually searched in the same way as NSW. The top layer, which is searched first, contains very few nodes, and deeper layers progressively include more and more nodes, with the bottom layer including all nodes. This means that the top layers contain longer hops across the vector space, permitting a sort of course-to-fine search. See above for an illustration.


## Constructing the Graph

The construction algorithm works as follows: we fix a number of layers, _L_, in advance. The value l=1 will correspond to the coarsest layer, where search begins, and l=L will correspond to the densest layer, where search finishes. We iterate through each vector to be inserted and sample an insertion layer following a truncated [geometric distribution](https://en.wikipedia.org/wiki/Geometric_distribution) (either rejecting _l > L_ or setting _l’ =_ min_(l, L)_). Say we sample _1 < l < L_ for the current vector. We perform a greedy search on the top layer, L, until we reach its local minimum. Then, we follow an edge from the local minimum in the _L_th layer to the corresponding vector in the _(L-1)_th layer and use it as the entry point to greedily search the _(L-1)_th layer.

This process is repeated until we reach the _l_th layer. We then begin to create nodes for the vector to be inserted, connecting it to its closest neighbors found by greedy search in the _l_th layer so far constructed, navigating to the _(l-1)_th layer and repeating until we have inserted the vector into the _1_st layer. An animation above makes this clear

We can see that this hierarchical graph construction method uses a clever explicit selection of the insertion node for each vector. We search the layers above the insertion layer constructed so far, searching efficiently from course-to-fine distances. Relatedly, the method separates links by different scales in each layer: the top layer affords long-scale hops across the search space, with the scale diminishing down to the bottom layer. Both of these modifications help avoid being trapped in sub-optimal minima and hasten search convergence at the cost of additional memory.


## Searching the Graph

The search procedure works much like the inner graph construction step. Starting from the top layer, we greedily navigate to the node or nodes closest to the query. Then we follow that node(s) down to the next layer and repeat the process. Our answer is obtained by the list of _R_ closest neighbors in the bottom layer, as illustrated by the animation above this.


## Conclusion

Vector databases like Milvus provide highly optimized and tuned implementations of HNSW, and it is often the best default search index for datasets that fit in memory.

We have sketched a high-level overview of how and why HNSW works, preferring visualizations and intuition over theory and mathematics. Consequently, we have omitted an exact description of the construction and search algorithms[[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); Alg 1-3], analysis of search and construction complexity [[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); §4.2], and less essential details like a heuristic for more effectively choosing neighbor nodes during construction [[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); Alg 5]. Moreover, we have omitted discussion of the algorithm’s hyperparameters, their meaning and how they affect the latency/speed/memory trade-off [[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); §4.1]. An understanding of this is important for using HNSW in practice.

The resources below contain further reading on these topics and a full Python pedagogical implementation (written by myself) for NSW and HNSW, including code to produce the animations in this article.


# Resources

- GitHub: “[HNSW-Illustrated: A small implementation of Hierarchical Navigable Small Worlds (HNSW), a vector search algorithm, for learning purposes](https://github.com/stefanwebb/hnsw-illustrated)”

- [HNSW | Milvus Documentation](https://milvus.io/docs/hnsw.md#HNSW)

- [Understanding Hierarchical Navigable Small Worlds (HNSW) - Zilliz Learn](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW)

- HNSW paper: “[Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://arxiv.org/abs/1603.09320)”

- NSW paper: “[Approximate nearest neighbor algorithm based on navigable small world graphs](https://publications.hse.ru/pubs/share/folder/x5p6h7thif/128296059.pdf)”


Understand HNSW for Vector Search


Understanding Hierarchical Navigable Small Worlds (HNSW) for Vector Search



## What is DiskANN?

[DiskANN](https://github.com/microsoft/DiskANN) represents a paradigm-shifting approach to [vector similarity search](https://zilliz.com/learn/vector-similarity-search). Before that, most vector index types like HNSW rely heavily on RAM to achieve low latency and high recall. While effective for moderate-sized datasets, this approach becomes prohibitively expensive and less scalable as data volumes grow. DiskANN offers a cost-effective alternative by leveraging SSDs to store the index, significantly reducing memory requirements.

DiskANN employs a flat graph structure optimized for disk access, allowing it to handle billion-scale datasets with a fraction of the memory footprint required by in-memory methods. For instance, DiskANN can index up to a billion vectors while achieving 95% search accuracy with 5ms latencies, whereas RAM-based algorithms peak at 100–200 million points for similar performance.


![](https://assets.zilliz.com/Vector_indexing_and_search_workflow_with_Disk_ANN_41cdf33652.png)

_Figure 1: Vector indexing and search workflow with DiskANN_

Although DiskANN may introduce slightly higher latency compared to RAM-based approaches, the trade-off is often acceptable given the substantial cost savings and scalability benefits. DiskANN is particularly suitable for applications requiring large-scale vector search on commodity hardware.

This article will explain the clever methods DiskANN has to leverage SSD in addition to RAM and reduce costly SSD reads. 


## How Does DiskANN Work?

DiskANN is a graph-based vector search method in the same family of methods as HNSW. We first construct a search graph where the nodes correspond to vectors (or groups of vectors), and edges denote that a pair of vectors is “relatively close” in some sense. A typical search randomly chooses an “entry node”, and navigates to its neighbor closest to the query, repeating in a greedy fashion until a local minimum is reached.

Graph-based indexing frameworks differ primarily in how they construct the search graph and perform search. And in this section, we will do a technical deep dive into the innovations of DiskANN for these steps and how they permit low-latency, low-memory performance. (See the above figure for a summary.)


### An Overview

We assume that the user has generated a set of document vector embeddings. The first step is to cluster the embeddings. A search graph for each cluster is constructed separately using the Vamana algorithm (explained in the next section), and the results are merged into a single graph. _The divide-and-conquer strategy for creating the final search graph significantly reduces memory usage without too greatly affecting search latency or recall._

![](https://assets.zilliz.com/How_Disk_ANN_stores_vector_index_across_RAM_and_SSD_d6564b087f.jpg)

_Figure 2: How DiskANN stores vector index across RAM and SSD_

Having produced the global search graph, it's stored on SSD together with the full-precision vector embeddings. A major challenge is to finish the search within a bounded number of SSD reads, since SSD access is expensive relative to RAM access. So, a few clever tricks are used to restrict the number of reads:

First, the Vamana algorithm incentivizes shorter paths between close nodes while capping the maximum number of neighbors of a node. Second, a fixed-size data structure is used to store each node’s embedding and its neighbors (see the above figure). What this means is that we can address a node’s metadata by simply multiplying the data structure size by the node’s index and using this as an offset while simultaneously fetching the node’s embedding. Third, due to how SSD works, we can fetch multiple nodes per read request - in our case, the neighbor nodes - reducing the number of read requests further.

Separately, we compress the embeddings using product quantization and store them in RAM. In doing so, we can fit billions-scale vector datasets into a memory that is feasible on a single machine for quickly calculating _approximate vector similarities_ without disk reads. This provides guidance for reducing the number of neighbor nodes to access next on the SSD. Importantly, however, the search decisions are made using the _exact vector similarities_, with the full embeddings retrieved from SSD, which ensures higher recall. To emphasize, there is an initial phase of search using quantized embeddings in memory, and a subsequent search on a smaller subset reading from SSD.

In this description, we have glossed over two important albeit involved steps: how to construct the graph, and how to search the graph - the two steps indicated by the red boxes above. Let’s examine each of these in turn.


### “Vamana” Graph Construction

![](https://assets.zilliz.com/Vamana_Graph_Construction_ecb4dab839.jpg)

_Figure: “Vamana” Graph Construction_

The DiskANN authors develop a novel method for constructing the search graph, which they call the Vamana algorithm. It initializes the search graph by randomly adding O(N) edges. This will result in a graph that is “well-connected”, although without any guarantees on greedy search convergence. It then prunes and reconnects the edges in an intelligent way to ensure there are sufficient long-range connections (see above figure). Allow us to elaborate:


#### Initialization

The search graph is initialized to a random directed graph where each node has R out-neighbors. We also calculate the medoid of the graph, that is, the point that has the minimum average distance to all other points. You can think of this as analogous to a centroid that is a member of the set of nodes.


#### Search for Candidates

After initialization, we iterate over the nodes, performing both adding and removing edges at each step. First, we run a search algorithm on the selected node, p, to generate a list of candidates. The search algorithm starts at the medoid and greedily navigates closer and closer to the selected node, adding the out-neighbors of the closest node found so far at each step. The list of L found nodes closest to p is returned. (If you’re not familiar with the concept, the medoid of a graph is the point that has the minimum average distance to all other points and acts as an analog of a centroid for graphs.)


#### Pruning and Adding Edges

The node’s candidate neighbors are sorted by distance, and for each candidate, the algorithm checks whether it is “too close” in direction to an already chosen neighbor. If so, it's pruned. This promotes angular diversity among neighbors, which empirically leads to better navigation properties. In practice, this means that a search starting from a random node can more quickly reach any target node by exploring a sparse set of long-range and local links.

After pruning edges, edges along the greedy search path to p are added. Two passes of pruning are performed, varying the distance threshold for pruning so that long-term edges are added in the second pass.


## What’s Next?

Subsequent work has been built upon DiskANN for additional improvements. One noteworthy example, known as [FreshDiskANN](https://arxiv.org/abs/2105.09613), modifies the method to allow for the easy updating of the index after construction. This search index, which provides an excellent tradeoff between performance criteria, is available in the [Milvus](https://milvus.io/docs/overview.md) vector database as the `DISKANN` index type. 

```python
# Prepare index parameters
index_params = client.prepare_index_params()

# Add DiskANN index
index_params.add_index(
    field_name="vector",
    index_type="DISKANN",
    metric_type="COSINE"
)

# Create collection with index
client.create_collection(
    collection_name="diskann_collection",
    schema=schema,
    index_params=index_params
)
```

You can even tune the DiskANN parameters, such as `MaxDegree` and  `BeamWidthRatio`: see [the documentation page](https://milvus.io/docs/disk_index.md#On-disk-Index) for more details.


## Resources

- [Milvus Documentation on using DiskANN](https://milvus.io/docs/disk_index.md#On-disk-Index)

- [“DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node”](https://suhasjs.github.io/files/diskann_neurips19.pdf)

- [“FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search”](https://arxiv.org/abs/2105.09613)


DiskANN Explained


Throughout this week, we've shared a range of exciting innovations in Milvus that push the boundaries of vector database technology: 

- [Vector Search in the Real World: How to Filter Efficiently Without Killing Recall ](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus ](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

Now, as we wrap up our Milvus Week series, I'm excited to give you a sneak peek of what's coming in Milvus 2.6—a crucial milestone in our 2025 product roadmap that's currently in development, and how these improvements will transform AI-powered search. This upcoming release brings together all these innovations and more across three critical fronts: **cost-efficiency optimization**, **advanced search capabilities**, and **a new architecture** that pushes vector search beyond 10 billion vector scale.

Let's dive into some of the key improvements you can expect when Milvus 2.6 arrives this June, starting with what might be the most immediately impactful: dramatic reductions in memory usage and cost, and ultra-fast performance. 

![](https://assets.zilliz.com/Milvus_2_6_Preview_72_Memory_Reduction_Without_Compromising_Recall_and_4x_Faster_Than_Elasticsearch_c607b644f1.png)


## Cost-Reduction: Slash Memory Usage While Boosting Performance

Relying on expensive memory presents one of the biggest obstacles to scaling vector search to billions of records. Milvus 2.6 will introduce several key optimizations that dramatically lower your infrastructure costs while improving performance.


### RaBitQ 1-bit Quantization: 72% Memory Reduction with 4× QPS and No Recall Loss

Memory consumption has long been the Achilles' heel of large-scale vector databases. While vector quantization isn't new, most existing approaches sacrifice too much search quality for memory savings. Milvus 2.6 will tackle this challenge head-on by introducing[ RaBitQ 1-bit quantization](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md) in production environments.

What makes our implementation special is the adjustable Refine optimization capability we're building. By implementing a primary index with RaBitQ quantization plus SQ4/SQ6/SQ8 Refine options, we've achieved an optimal balance between memory usage and search quality (~95% recall).

Our preliminary benchmarks reveal promising results:

| **Performance** **Metric** | **Traditional IVF_FLAT** | **RaBitQ (1-bit) Only** | **RaBitQ (1-bit) + SQ8 Refine** |
| -------------------------- | ------------------------- | ----------------------- | ------------------------------- |
| Memory Footprint           | 100% (baseline)           | 3% (97% reduction)      | 28% (72% reduction)             |
| Recall Quality             | 95.2%                     | 76.3%                   | 94.9%                           |
| Query Throughput (QPS)     | 236                       | 648 (2.7× faster)       | 946 (4× faster)                 |

_Table: VectorDBBench evaluation with 1M vectors of 768 dimensions, tested on AWS m6id.2xlarge_

The real breakthrough here isn't just the memory reduction, but achieving this while simultaneously delivering a 4× throughput improvement without compromising accuracy. This means you'll be able to serve the same workload with 75% fewer servers or handle 4× more traffic on your existing infrastructure.

For enterprise users using fully managed Milvus on[ Zilliz Cloud](https://zilliz.com/cloud), we're developing automated configuration profiles that will dynamically adjust RaBitQ parameters based on your specific workload characteristics and precision requirements.


### 400% Faster Full-text Search Than Elasticsearch

[Full-text search](https://milvus.io/blog/full-text-search-in-milvus-what-is-under-the-hood.md) capabilities in vector databases have become essential for building hybrid retrieval systems. Since introducing BM25 in [Milvus 2.5](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md), we've received enthusiastic feedback—along with requests for better performance at scale.

Milvus 2.6 will deliver substantial performance gains on BM25. Our testing on the BEIR dataset shows 3-4× higher throughput than Elasticsearch with equivalent recall rates. For some workloads, the improvement reaches up to 7× higher QPS.

![](https://assets.zilliz.com/Figure_Milvus_vs_Elasticsearch_on_throughput_140b7c1b06.png)

Figure: Milvus vs. Elasticsearch on throughput 


### JSON Path Index: 99% Lower Latency for Complex Filtering

Modern AI applications rarely rely on vector similarity alone—they almost always combine vector search with metadata filtering. As these filtering conditions become more complex (especially with nested JSON objects), query performance can deteriorate rapidly.

Milvus 2.6 will introduce a targeted indexing mechanism for nested JSON paths that allows you to create indexes on specific paths (e.g., `$meta user_info.location`) within JSON fields. Instead of scanning entire objects, Milvus will directly look up values from pre-built indexes.

In our evaluation with 100 M+ records, JSON Path Index reduced filter latency from **140ms** (P99: 480ms) to just **1.5ms** (P99: 10ms)—a 99% reduction that will transform previously impractical queries into instant responses.

This feature will be particularly valuable for:

- Recommendation systems with complex user attribute filtering

- RAG applications that filter documents by various labels


## Next-Generation Search: From Basic Vector Similarity to Production-Grade Retrieval

Vector search alone isn't enough for modern AI applications. Users demand the precision of traditional information retrieval combined with the semantic understanding of vector embeddings. Milvus 2.6 will introduce several advanced search features that bridge this gap.


### Better Full-text Search with Multi-language Analyzer

Full-text search is highly language-dependent.. Milvus 2.6 will introduce a completely revamped text analysis pipeline with multi-language support:

- `RUN_ANALYZER` syntax support for analyzer/tokenization configuration observability

- Lindera tokenizer for Asian languages like Japanese and Korean

- ICU tokenizer for comprehensive multilingual support

- Granular language configuration for defining language-specific tokenization rules

- Enhanced Jieba with support for custom dictionary integration

- Expanded filter options for more precise text processing

For global applications, this means better multilingual search without specialized per-language indexing or complex workarounds.


### Phrase Match: Capturing Semantic Nuance in Word Order

Word order conveys critical meaning distinctions that keyword search often misses. Try comparing "machine learning techniques" with "learning machine techniques"—same words, totally different meaning.

Milvus 2.6 will add **Phrase Match**, giving users more control over word order and proximity than full-text search or exact string match:

```
PHRASE_MATCH(field_name, phrase, slop)
```


The `slop` parameter will provide flexible control over word proximity—0 requires exact consecutive matches, while higher values allow for minor variations in phrasing.

This feature will be particularly valuable for:

- Legal document search where exact phrasing carries legal significance

- Technical content retrieval where term order distinguishes different concepts

- Patent databases where specific technical phrases must be matched precisely


### Time-Aware Decay Functions: Automatically Prioritize Fresh Content

Information value often diminishes with time. News articles, product releases, and social posts all become less relevant as they age, yet traditional search algorithms treat all content equally, regardless of timestamp.

Milvus 2.6 will introduce **Decay Functions** for time-aware ranking that automatically adjust relevance scores based on document age.

![](https://assets.zilliz.com/decay_function_210e65f9a0.png)

You'll be able to configure:

- **Function type**: Exponential (rapid decay), Gaussian (gradual decay), or Linear (constant decay)

- **Decay rate**: How quickly relevance diminishes over time

- **Origin point**: The reference timestamp for measuring time differences

This time-sensitive re-ranking will ensure that the most up-to-date and contextually relevant results appear first, which is crucial for news recommendation systems, e-commerce platforms, and social media feeds.


### Data in, Data Out: From Raw Text to Vector Search in One Step

One of the biggest developer pain points with vector databases has been the disconnect between raw data and vector embeddings. Milvus 2.6 will dramatically simplify this workflow with a new **Function** interface that integrates third-party embedding models directly into your data pipeline. This streamlines your vector search pipeline with one single call. 

Instead of pre-computing embeddings, you'll be able to:

1. **Insert raw data directly**: Submit text, images, or other content to Milvus

2. **Configure embedding providers for vectorization**: Milvus can connect to embedding model services like OpenAI, AWS Bedrock, Google Vertex AI, and Hugging Face.

3. **Query using natural language**: Search using text queries, not vector embeddings

This will create a streamlined "Data-In, Data-Out" experience where Milvus handles the vector generation internally, making your application code more straightforward.


## Architectural Evolution: Scaling to Hundreds of Billions of Vectors

A good database doesn’t just have great features, it must also deliver those features at scale, battle-tested in production. 

Milvus 2.6 will introduce a fundamental architectural change that enables cost-effective scaling to hundreds of billions of vectors. The highlight is a new hot-cold tiered storage architecture that intelligently manages data placement based on access patterns, automatically moving hot data to high-performance memory/SSD while placing cold data in more economical object storage. This approach can dramatically reduce costs while maintaining query performance where it matters most. 

Additionally, a new [Streaming Node](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md#StreamingService-Built-for-Real-Time-Data-Flow) will enable real-time vector processing with direct integration to streaming platforms like Kafka and Pulsar and the newly created [Woodpecker](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md), making new data searchable immediately without batch delays.


## Stay tuned for Milvus 2.6 

Milvus 2.6 is currently in active development and will be available this June. We're excited to bring you these breakthrough performance optimizations, advanced search capabilities, and a new architecture to help you build scalable AI applications at lower cost.

In the meantime, we welcome your feedback on these upcoming features. What excites you most? Which capabilities would have the most impact on your applications? Join the conversation in our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or follow our progress on[ GitHub](https://github.com/milvus-io/milvus).

Want to be the first to know when Milvus 2.6 is released? Follow us on[ LinkedIn](https://www.linkedin.com/company/zilliz/) or[ X](https://twitter.com/milvusio) for the latest updates.


Milvus 2.6 Preview: 72% Memory Reduction Without Compromising Recall and 4x Faster Than Elasticsearch



Large Language Models (LLMs) have transformed the AI landscape with their ability to write code, create content, and solve complex problems. However, these powerful models require enormous amounts of high-quality data to fuel their training.

The challenge is that raw training data often contains significant redundancy. It's like teaching a child by repeating the same lessons over and over while skipping other important topics. A large AI company approached us with precisely this problem - they were building an ambitious new language model but struggled with deduplicating tens of billions of documents. Traditional matching methods couldn't scale to this volume, and specialized deduplication tools required massive computational resources, making them economically unviable.

To solve this problem, we introduced MinHash LSH (Locality Sensitive Hashing) indexing in Milvus 2.6. This article will explore how MinHash LSH efficiently solves the data deduplication problem for LLM training.


![](https://assets.zilliz.com/Chat_GPT_Image_May_16_2025_09_46_39_PM_1f3290ce5e.png)


## Data Deduplication: Why It Matters for LLM Training

High-quality, diverse data is essential for training powerful LLMs. When duplicate content appears in training data, it creates several significant problems:

- **Wasted Resources:** Redundant data increases training time, costs, and energy consumption.

- **Reduced Performance:** Models can overfit to repeated content, limiting their ability to generalize to new information.

- **Memorization Effect:** Duplicated content increases the chance of models memorizing and reproducing specific text verbatim. It could also potentially lead to privacy leaks or copyright issues.

- **Misleading Evaluations:** Duplicates between training and test sets can accidentally inflate performance metrics.

There are three main approaches to finding and removing duplicates:

- **Exact Matching:** Identifies identical duplicates through hashing.

- **Approximate Matching:** Finds near-duplicates using algorithms like MinHash LSH and Jaccard similarity.

- **Semantic Matching:** Identifies content with similar meaning using vector embeddings.

With pre-training corpora reaching terabytes or even petabytes, traditional exact matching methods like pairwise comparisons are computationally infeasible. Semantic deduplication adds significant overhead by using embedding models to generate vectors. We need more innovative approximate methods—like **MinHash LSH**—that balance recall and precision while keeping costs manageable, making large-scale deduplication practical.


## MinHash LSH: Efficiently Detecting Near-Duplicates in Massive Datasets

To find near-duplicates in an ocean of training data, we need an approximate matching algorithm that’s both efficient and accurate. MinHash LSH (Locality Sensitive Hashing) is a great tool for this goal. Let’s break down this seemingly complex term step by step. 


### Step 1: Representing Documents with MinHash

First, we need a way to measure document similarity. The standard approach uses Jaccard similarity:

$$J(A,B) = \frac{|A\cap B|}{|A \cup B|}$$

This formula measures the overlap between document A and document B - specifically, the ratio of shared elements to total unique elements. A higher value means the documents are more similar.

However, computing this directly for billions of document pairs is resource-intensive and would take years. MinHash creates compact "fingerprints" (signatures) that preserve similarity relationships while making comparisons much faster. 

1. **Shingling:** Break each document into overlapping sequences of words or characters (k-shingles). For example, the sentence “I love vector search” with k=3 (by word) yields: {“I love vector”, “love vector search”}

![](https://assets.zilliz.com/shingling_858ad58efa.png)

2. **MinHashing:** Apply multiple hash functions to each set of shingles and record the minimum hash value from each function. This results in a signature vector for each document.

![](https://assets.zilliz.com/minhash_041003210a.png)

When calculating similarity, the probability that hash values align at the same positions in the MinHash signatures of two documents (which corresponds to the Jaccard distance of these signatures) provides a close approximation of the Jaccard similarity of their original shingle sets. This allows us to effectively estimate document similarity without directly comparing the larger original texts; instead, we can analyze their compact MinHash signatures.  

The MinHash principle involves using the word with the smallest hash value to represent the entire document, enhancing accuracy by incorporating additional hash functions. Minor word changes are likely to be overlooked as they typically do not affect the minimum hash value. In contrast, more substantial changes tend to alter the hash value and are more easily detected. This method can be seen as a min-pooling of hash values across various words. In addition to MinHash, alternatives like SimHash are available for generating document signatures, but those will not be discussed here.


### Step 2: Identifying Similar Documents via LSH

Even with compact MinHash signatures, comparing every pair across millions or billions of documents remains computationally expensive. That’s where **Locality Sensitive Hashing (LSH)** comes in.

The key idea of LSH is to use hash functions that **intentionally cause collisions**—similar items are more likely to hash into the same bucket, while dissimilar ones are not. This is the opposite of traditional hashing, which aims to avoid collisions.

For MinHash, a popular LSH strategy is the **banding technique**:

1. **Banding**: Split each MinHash signature (a vector of length _N_) into _b_ bands, each with _r_ dims (_N = b × r_).

2. **Hashing Bands:** Hash each band (a sub-vector of _r_ values) into a bucket using a standard hash function.

3. **Candidate Pairs:** If two documents share a bucket in **any** band, they are flagged as potential matches.

By adjusting the number of bands (b) and the number of dimensions per band (r), you can control the trade-off between recall, precision, and search efficiency. 

The key idea is: highly similar documents will have many matching hash values in their MinHash signatures. When these signatures are split into bands, even one band with all matching values is enough to place two documents in the same bucket. The more similar the documents are, the higher the probability that this happens in at least one band, allowing LSH to efficiently surface candidate pairs without exhaustively comparing all signatures.

In short, **MinHash + LSH** enables scalable approximate deduplication: MinHash compresses documents into compact signatures, and LSH efficiently narrows the search space by grouping likely matches. It’s like spotting twins in a crowd: first, take a quick feature snapshot of everyone (MinHash), group lookalikes (LSH), then inspect the smaller groups closely for actual duplicates.


## Integrating MinHash LSH in Milvus 2.6

A real-world need drove the integration of MinHash LSH into Milvus 2.6. As mentioned earlier, a Milvus user—one of the leading LLM companies—approached us with a challenge: efficiently deduplicating massive volumes of text data for LLM pre-training. 

Traditional deduplication pipelines typically rely on external tools decoupled from storage and retrieval systems, requiring costly data transfers between components. This fragmented workflow increases operational overhead and prevents full utilization of distributed computing resources.

Recognizing Milvus’s strengths in handling high-throughput vector data, a natural idea emerged: **_What if MinHash LSH were built into Milvus natively, making approximate deduplication a first-class database feature?_**

This approach enables a complete workflow from deduplication to semantic retrieval within Milvus, simplifying MLOps while leveraging its scalability and unified API. Together with our partner, we optimized MinHash LSH for Milvus’s cloud-native architecture, resulting in a fast and scalable solution for large-scale deduplication.


### Core capabilities in Milvus 2.6 include:

- **Native MinHash LSH Indexing:** Implements the standard banding technique for LSH and supports optional Jaccard re-ranking to improve recall. Provides both in-memory and mmap-based implementations for flexibility across different workloads.

- **Seamless API Integration:** Users can define MinHash vector fields, build `MINHASH_LSH` indexes, insert signature data, and perform approximate similarity searches using Milvus’s standard SDK and declarative APIs.

- **Distributed and Scalable:** Built on Milvus’s cloud-native architecture, the feature supports horizontal scaling for large datasets and high-throughput processing.

This integration delivered impressive results. By running MinHash LSH on fully-managed Milvus ([Zilliz Cloud](https://zilliz.com/cloud)), we helped this user deduplicate **10 billion documents** efficiently. Compared to their previous MapReduce-based approach, the new solution **more than doubled processing speed** and achieved **3-5x cost savings**, thanks to Milvus’s optimized indexing and query execution.


## Hands-On: Deduplicating LLM Datasets Using Milvus

Let's roll up our sleeves and use MinHash LSH in Milvus 2.6 to perform approximate deduplication at scale.


### Prerequisite: Generating MinHash Signatures

Milvus handles the indexing and search of **pre-generated** MinHash signatures. You’ll need to generate these during preprocessing using tools like `datasketch` in Python or a custom implementation. The typical steps are:

1. Read raw documents

2. Shingle (tokenize or chunk) each document

3. Apply multiple hash functions to generate a MinHash signature (e.g., an uint64 array of size 128 )

```
from datasketch import MinHash

text = "example text for minhash signature"
tokens = text.lower().split()  # Step 1 & 2: preprocess + tokenize/shingle
mh = MinHash(num_perm=128)     # Step 3: initialize MinHash
for token in tokens:
    mh.update(token.encode('utf-8'))  # Add shingles to MinHash
signature = mh.hashvalues  # This is your MinHash signature (128-dimensional)
```



### Step 1: Create a Schema in Milvus

We need to create a Milvus collection to store the MinHash signatures and their corresponding document IDs.

```
import numpy as np
from pymilvus import MilvusClient
from pymilvus import DataType
MILVUS_URI = "localhost:19530"
COLLECTION_NAME = "llm_data_dedup_minhash"
MINHASH_DIM = 128
MINHASH_BIT_WIDTH = 64 # Assuming 64-bit hash values

# Load data from NPY file
base = np.load('minhash_vectors.npy')
ids = [str(i) for i in range(base.shape[0])]  # Generate string IDs

client = MilvusClient(uri=MILVUS_URI)
# Check and drop existing collection if needed
if collection_name in client.list_collections():
    print(f"Collection {collection_name} exists, dropping it...")
    client.drop_collection(collection_name)
# Create collection schema
schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=False,
)

schema.add_field(field_name="input_id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="minhash", datatype=DataType.BINARY_VECTOR, dim=MINHASH_DIM * MINHASH_BIT_WIDTH)
schema.add_field(field_name="id", datatype=DataType.VARCHAR, max_length=200)
```


### **Step 2: Create the MINHASH_LSH Index and Collection**

This is the core step. We need to specify JACCARD as the metric type and configure LSH-related parameters. 

```
INDEX_FIELD_NAME = "minhash_signature"
# Metric type, should be JACCARD for MinHash LSH
METRIC_TYPE = "MHJACCARD"
INDEX_TYPE = "MINHASH_LSH"
MINHASH_DIM = 128
MINHASH_BIT_WIDTH = 64 # Assuming 64-bit hash values

index_params = MilvusClient.prepare_index_params()

index_params.add_index(
    field_name=INDEX_FIELD_NAME,
    index_type=INDEX_TYPE,
    metric_type=METRIC_TYPE,
    params={
        # LSH-specific parameters might be configured here, e.g.:
        # "band": 32, # Hypothetical parameter: number of bands
        # "element_bit_width": 64 # Bit width of minhash values
    }
)
# Create collection
client.create_collection(
    collection_name=collection_name,
    schema=schema,
    index_params=index_params
)
```


A Note on Parameter Tuning: The effectiveness of MinHash LSH heavily depends on parameter choices. For instance, the number of hash functions used during MinHash signature generation (i.e., `MINHASH_DIM`) affects the signature's precision and size. In the LSH phase, the number of bands (`num_bands`) and rows per band together determine the sensitivity range of the similarity threshold and the balance between recall and precision. Users need to experiment and fine-tune based on their dataset characteristics and deduplication requirements. This is often an iterative process.


### **Step 3: Insert MinHash Signatures**

Let's say you have a batch of documents and their corresponding MinHash signatures.

```
# Insert data in batches
batch_size = 2000
total_records = base.shape[0]
num_batches = (total_records + batch_size - 1) // batch_size

for batch_idx in range(num_batches):
    start = batch_idx * batch_size
    end = min((batch_idx + 1) * batch_size, total_records)
    
    print(f"Inserting batch {batch_idx + 1}/{num_batches} (records {start}-{end})")
    
    # Prepare batch data
    batch_data = [{
        "input_id": i,
        "minhash": base[i].tobytes(),
        "id": ids[i]
    } for i in range(start, end)]
    
    # Insert batch
    client.insert(collection_name, batch_data)

print("Data insertion complete")
```


### Step 5: Search for Near-Duplicates

Use a document’s MinHash signature to search for similar documents in the collection.

```
# Perform search
search_vectors = [vec.tobytes() for vec in base[:10]]
results = client.search(
    collection_name, 
    data=search_vectors, 
    param={"metric_type": METRIC_TYPE, "params": search_params_lsh},
    limit=1, 
    output_fields=["id"])

print("\nSearch results:")
for i, result in enumerate(results):
    print(f"Query {i}:")
    for hit in result:
        print(f"  - ID: {hit['entity']['id']}, Distance: {hit['distance']}")
```



### Step 6: Post-Processing and Clustering

The returned results are **candidate near-duplicates**. To form complete deduplication groups, you can apply clustering techniques like **Union-Find** on the candidate pairs. Each resulting group represents a set of duplicates; keep one representative document and archive or remove the rest.


## **Conclusion**

MinHash LSH in Milvus 2.6 is a leap forward in AI data processing. What started as a solution for LLM data deduplication now opens doors to broader use cases—web content cleanup, catalog management, plagiarism detection, and more.

## Getting Started with Milvus 2.6

Milvus 2.6 is available now. In addition to MinHash LSH, it introduces dozens of new features and performance optimizations such as tiered storage, RabbitQ quantization method, and enhanced full-text search and multitenancy, directly addressing the most pressing challenges in vector search today: scaling efficiently while keeping costs under control.

Ready to explore everything Milvus offers? Dive into our[ release notes](https://milvus.io/docs/release_notes.md), browse the[ complete documentation](https://milvus.io/docs), or check out our[ feature blogs](https://milvus.io/blog). 

If you have any questions or have a similar use case, feel free to reach out to us through our [Discord community](https://discord.com/invite/8uyFbECzPX) or file an issue on[ GitHub](https://github.com/milvus-io/milvus) — we're here to help you make the most of Milvus 2.6.






MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data



**TL;DR:** We built Woodpecker, a cloud-native Write-Ahead Logging (WAL) system, to replace Kafka and Pulsar in Milvus 2.6. The result? Simplified operations, better performance, and lower costs for our Milvus vector database.

![](https://assets.zilliz.com/We_Replaced_Kafka_Pulsar_with_a_Woodpecker_for_Milvus_Here_s_What_Happened_77e8de27a9.png)

## The Starting Point: When Message Queues No Longer Fit

We loved and used Kafka and Pulsar. They worked until they didn't. As Milvus, the leading open-source vector database, evolved, we found that these powerful message queues no longer met our scalability requirements. So we made a bold move: we rewrote the streaming backbone in Milvus 2.6 and implemented our own WAL — **Woodpecker**.

Let me walk you through our journey and explain why we made this change, which might seem counterintuitive at first glance.


## Cloud-Native From Day One

Milvus has been a cloud-native vector database from its inception. We leverage Kubernetes for elastic scaling and quick failure recovery, alongside object storage solutions like Amazon S3 and MinIO for data persistence.

This cloud-first approach offers tremendous advantages, but it also presents some challenges:

- Cloud object storage services like S3 provide virtually unlimited capability of handling throughputs and availability, but with latencies often exceeding 100ms.

- These services’ pricing models (based on access patterns and frequency) can add unexpected costs to real-time database operations.

- Balancing cloud-native characteristics with the demands of real-time vector search introduces significant architectural challenges.


## The Shared Log Architecture: Our Foundation

Many vector search systems restrict themselves to batch processing because building a streaming system in a cloud-native environment presents even greater challenges. In contrast, Milvus prioritizes real-time data freshness and implements a shared log architecture—think of it as a hard drive for a filesystem.

This shared log architecture provides a critical foundation that separates consensus protocols from core database functionality. By adopting this approach, Milvus eliminates the need to manage complex consensus protocols directly, allowing us to focus on delivering exceptional vector search capabilities.

We're not alone in this architectural pattern—databases such as AWS Aurora, Azure Socrates, and Neon all leverage a similar design. **However, a significant gap remains in the open-source ecosystem: despite the clear advantages of this approach, the community lacks a low-latency, scalable, and cost-effective distributed write-ahead log (WAL) implementation.**

Existing solutions like Bookie proved inadequate for our needs due to their heavyweight client design and the absence of production-ready SDKs for Golang and C++. This technological gap led us to our initial approach with message queues.


## Our Initial Solution: Message Queues as WAL

To bridge this gap, our initial approach utilized message queues (Kafka/Pulsar) as our write-ahead log (WAL). The architecture worked like this: 

- All incoming real-time updates flow through the message queue.

- Writers receive immediate confirmation once it is accepted by the message queue.

- QueryNode and DataNode process this data asynchronously, ensuring high write throughput while maintaining data freshness

![](https://assets.zilliz.com/Figure_Milvus_2_0_Architecture_Overview_465f5ba27a.png)

Figure: Milvus 2.0 Architecture Overview

This system effectively provided immediate write confirmation while enabling asynchronous data processing, which was crucial for maintaining the balance between throughput and data freshness that Milvus users expect.


## Why We Needed Something Different for WAL

With Milvus 2.6, we've decided to phase out external message queues in favor of Woodpecker, our purpose-built, cloud-native WAL implementation. This wasn't a decision we made lightly. After all, we had successfully used Kafka and Pulsar for years.

The issue wasn't with these technologies themselves—both are excellent systems with powerful capabilities. Instead, the challenge came from the increasing complexity and overhead that these external systems introduced as Milvus evolved. As our requirements became more specialized, the gap between what general-purpose message queues offered and what our vector database needed continued to widen.

Three specific factors ultimately drove our decision to build a replacement:


### Operational Complexity

External dependencies like Kafka or Pulsar demand dedicated machines with multiple nodes and careful resource management. This creates several challenges:

- Increased operational complexity

* Steeper learning curves for system administrators

- Higher risks of configuration errors and security vulnerabilities


### Architectural Constraints

Message queues like Kafka have inherent limitations on the number of supported topics. We developed VShard as a workaround for topic sharing across components, but this solution—while effectively addressing scaling needs—introduced significant architectural complexity.

These external dependencies made it harder to implement critical features—such as log garbage collection—and increased integration friction with other system modules. Over time, the architectural mismatch between general-purpose message queues and the specific, high-performance demands of a vector database became increasingly clear, prompting us to reassess our design choices.


### Resource Inefficiency

Ensuring high availability with systems like Kafka and Pulsar typically demands:

- Distributed deployment across multiple nodes

- Substantial resource allocation even for smaller workloads

- Storage for ephemeral signals (like Milvus's Timetick), which don’t actually require long-term retention

However, these systems lack the flexibility to bypass persistence for such transient signals, leading to unnecessary I/O operations and storage usage. This leads to disproportionate resource overhead and increased cost—especially in smaller-scale or resource-constrained environments.


## Introducing Woodpecker - A Cloud-Native, High-Performance WAL Engine

In Milvus 2.6, we’ve replaced Kafka/Pulsar with **Woodpecker**, a purpose-built, cloud-native WAL system. Designed for object storage, Woodpecker simplifies operations while boosting performance and scalability.

Woodpecker is built from the ground up to maximize the potential of cloud-native storage, with a focused goal: to become the highest-throughput WAL solution optimized for cloud environments while delivering the core capabilities needed for an append-only write-ahead log.


### The Zero-Disk Architecture for Woodpecker 

Woodpecker's core innovation is its **Zero-Disk architecture**:

- All log data stored in cloud object storage (such as Amazon S3, Google Cloud Storage, or Alibaba OS)

- Metadata managed through distributed key-value stores like etcd

- No local disk dependencies for core operations

![](https://assets.zilliz.com/Figure_Woodpecker_Architecture_cc31e15ed9.png)

Figure:  Woodpecker Architecture Overview

This approach dramatically reduces operational overhead while maximizing durability and cloud efficiency. By eliminating local disk dependencies, Woodpecker aligns perfectly with cloud-native principles and significantly reduces the operational burden on system administrators.


### Performance Benchmarks: Exceeding Expectations

We ran comprehensive benchmarks to evaluate Woodpecker's performance in a single-node, single-client, single-log-stream setup. The results were impressive when compared to Kafka and Pulsar:

| **System** | **Kafka**   | **Pulsar** | **WP MinIO** | **WP Local** | **WP S3** |
| ---------- | ----------- | ---------- | ------------ | ------------ | --------- |
| Throughput | 129.96 MB/s | 107 MB/s   | 71 MB/s      | 450 MB/s     | 750 MB/s  |
| Latency    | 58 ms       | 35 ms      | 184 ms       | 1.8 ms       | 166 ms    |

For context, we measured the theoretical throughput limits of different storage backends on our test machine:

- **MinIO**: ~110 MB/s

- **Local file system**: 600–750 MB/s

- **Amazon S3 (single EC2 instance)**: up to 1.1 GB/s

Remarkably, Woodpecker consistently achieved 60-80% of the maximum possible throughput for each backend—an exceptional efficiency level for middleware.


#### Key Performance Insights

1. **Local File System Mode**: Woodpecker achieved 450 MB/s—3.5× faster than Kafka and 4.2× faster than Pulsar—with ultra-low latency at just 1.8 ms, making it ideal for high-performance single-node deployments.

2. **Cloud Storage Mode (S3)**: When writing directly to S3, Woodpecker reached 750 MB/s (about 68% of S3's theoretical limit), 5.8× higher than Kafka and 7× higher than Pulsar. While latency is higher (166 ms), this setup provides exceptional throughput for batch-oriented workloads.

3. **Object Storage Mode (MinIO)**: Even with MinIO, Woodpecker achieved 71 MB/s—around 65% of MinIO's capacity. This performance is comparable to Kafka and Pulsar but with significantly lower resource requirements.

Woodpecker is particularly optimized for concurrent, high-volume writes where maintaining order is critical. And these results only reflect the early stages of development—ongoing optimizations in I/O merging, intelligent buffering, and prefetching are expected to push performance even closer to theoretical limits.


### Design Goals

Woodpecker addresses the evolving demands of real-time vector search workloads through these key technical requirements:

- High-throughput data ingestion with durable persistence across availability zone

- Low-latency tail reads for real-time subscriptions and high-throughput catch-up reads for failure recovery

- Pluggable storage backends, including cloud object storage and file systems with NFS protocol support

- Flexible deployment options, supporting both lightweight standalone setups and large-scale clusters for multi-tenant Milvus deployments


### Architecture Components

A standard Woodpecker deployment includes the following components. 

- **Client** – Interface layer for issuing read and write requests

- **LogStore** – Manages high-speed write buffering, asynchronous uploads to storage, and log compaction

- **Storage Backend** – Supports scalable, low-cost storage services such as S3, GCS, and file systems like EFS

- **ETCD** – Stores metadata and coordinates log state across distributed nodes


### Flexible Deployments to Match Your Specific Needs 

Woodpecker offers two deployment modes to match your specific needs:

**MemoryBuffer Mode – Lightweight and Maintenance-Free**

MemoryBuffer Mode provides a simple and lightweight deployment option where Woodpecker temporarily buffers incoming writes in memory and periodically flushes them to a cloud object storage service. Metadata is managed using etcd to ensure consistency and coordination. This mode is best suited for batch-heavy workloads in smaller-scale deployments or production environments that prioritize simplicity over performance, especially when low write latency is not critical.

![](https://assets.zilliz.com/Figure_The_memory_Buffer_Mode_3429d693a1.png)

_Figure: The memoryBuffer Mode_ 

**QuorumBuffer Mode – Optimized for Low-Latency, High-Durability Deployments**

QuorumBuffer Mode is designed for latency-sensitive, high-frequency read/write workloads requiring both real-time responsiveness and strong fault tolerance. In this mode, Woodpecker functions as a high-speed write buffer with three-replica quorum writes, ensuring strong consistency and high availability.

A write is considered successful once it's replicated to at least two of the three nodes, typically completing within single-digit milliseconds, after which the data is asynchronously flushed to cloud object storage for long-term durability. This architecture minimizes on-node state, eliminates the need for large local disk volumes, and avoids complex anti-entropy repairs often required in traditional quorum-based systems.

The result is a streamlined, robust WAL layer ideal for mission-critical production environments where consistency, availability, and fast recovery are essential.

![](https://assets.zilliz.com/Figure_The_Quorum_Buffer_Mode_72573dc666.png)

_Figure: The QuorumBuffer Mode_ 


## StreamingService: Built for Real-Time Data Flow

Beyond Woodpecker, Milvus 2.6 introduces the **StreamingService**—a specialized component designed for log management, log ingestion, and streaming data subscription.

To understand how our new architecture works, it's important to clarify the relationship between these two components:

- **Woodpecker** is the storage layer that handles the actual persistence of write-ahead logs, providing durability and reliability

- **StreamingService** is the service layer that manages log operations and provides real-time data streaming capabilities

Together, they form a complete replacement for external message queues. Woodpecker provides the durable storage foundation, while StreamingService delivers the high-level functionality that applications interact with directly. This separation of concerns allows each component to be optimized for its specific role while working seamlessly together as an integrated system. 


### Adding Streaming Service to Milvus 2.6 

![](https://assets.zilliz.com/Figure_Milvus_2_6_Architecture_Overview_238428c58f.png)

Figure: Streaming Service Added in Milvus 2.6 Architecture

The Streaming Service is composed of three core components: 

**Streaming Coordinator**

- Discovers available Streaming Nodes by monitoring Milvus ETCD sessions

- Manages the status of WALs and collects load balancing metrics through the ManagerService

**Streaming Client**

- Queries the AssignmentService to determine WAL segment distribution across Streaming Nodes

- Performs read/write operations via the HandlerService on the appropriate Streaming Node

**Streaming Node**

- Handles actual WAL operations and provides publish-subscribe capabilities for real-time data streaming

- Includes the **ManagerService** for WAL administration and performance reporting

- Features the **HandlerService** that implements efficient publish-subscribe mechanisms for WAL entries

This layered architecture allows Milvus to maintain clear separation between the streaming functionality (subscription, real-time processing) and the actual storage mechanisms. Woodpecker handles the "how" of log storage, while StreamingService manages the "what" and "when" of log operations.

As a result, the Streaming Service significantly enhances the real-time capabilities of Milvus by introducing native subscription support, eliminating the need for external message queues. It reduces memory consumption by consolidating previously duplicated caches in the query and data paths, lowers latency for strongly consistent reads by removing asynchronous synchronization delays, and improves both scalability and recovery speed across the system.


## Conclusion - Streaming on a Zero-Disk Architecture

Managing state is hard. Stateful systems often sacrifice elasticity and scalability. The increasingly accepted answer in cloud-native design is to decouple state from compute—allowing each to scale independently.

Rather than reinventing the wheel, we delegate the complexity of durable, scalable storage to the world-class engineering teams behind services like AWS S3, Google Cloud Storage, and MinIO. Among them, S3 stands out for its virtually unlimited capacity, eleven nines (99.999999999%) of durability, 99.99% availability, and high-throughput read/write performance.

But even "zero-disk" architectures have trade-offs. Object stores still struggle with high write latency and small-file inefficiencies—limitations that remain unresolved in many real-time workloads.

For vector databases—especially those supporting mission-critical RAG, AI agents, and low-latency search workloads—real-time access and fast writes are non-negotiable. That's why we rearchitected Milvus around Woodpecker and the Streaming Service. This shift simplifies the overall system (let's face it—no one wants to maintain a full Pulsar stack inside a vector database), ensures fresher data, improves cost-efficiency, and speeds up failure recovery.

We believe Woodpecker is more than just a Milvus component—it can serve as a foundational building block for other cloud-native systems. As cloud infrastructure evolves, innovations like S3 Express may bring us even closer to the ideal: cross-AZ durability with single-digit millisecond write latency.

## Getting Started with Milvus 2.6

Milvus 2.6 is available now. In addition to Woodpecker, it introduces dozens of new features and performance optimizations such as tiered storage, RabbitQ quantization method, and enhanced full-text search and multitenancy, directly addressing the most pressing challenges in vector search today: scaling efficiently while keeping costs under control.

Ready to explore everything Milvus offers? Dive into our[ release notes](https://milvus.io/docs/release_notes.md), browse the[ complete documentation](https://milvus.io/docs), or check out our[ feature blogs](https://milvus.io/blog).

Have questions? You're also welcome to join our [Discord community](https://discord.com/invite/8uyFbECzPX) or file an issue on[ GitHub](https://github.com/milvus-io/milvus) — we're here to help you make the most of Milvus 2.6.


 


We Replaced Kafka/Pulsar with a Woodpecker for Milvus


We Replaced Kafka/Pulsar with a Woodpecker for Milvus—Here’s What Happened




## The Vector Database You Chose Based on Benchmarks Might Fail in Production


When selecting a [vector database](https://milvus.io/blog/what-is-a-vector-database.md) for your AI application, conventional benchmarks are like test-driving a sports car on an empty track, only to find it stalls in rush hour traffic. The uncomfortable truth? Most benchmarks only evaluate performance in artificial conditions that never exist in production environments.

![](https://assets.zilliz.com/Benchmarks_Lie_Vector_D_Bs_Deserve_a_Real_Test_9280c66efc.png)

Most benchmarks test vector databases **after** all data has been ingested and the index is fully built. But in production, data never stops flowing. You don’t get to pause your system for hours to rebuild an index.

We’ve seen the disconnect firsthand. For example, Elasticsearch might boast millisecond-level query speeds, but behind the scenes, we’ve watched it take **over 20 hours** just to optimize its index. That’s downtime no production system can afford, especially in AI workloads that demand continuous updates and instant responses.

With Milvus, after running countless Proof of Concept (PoC) evaluations with enterprise clients, we've uncovered a troubling pattern: **vector databases that excel in controlled lab environments frequently struggle under actual production loads.** This critical gap doesn't just frustrate infrastructure engineers—it can derail entire AI initiatives built on these misleading performance promises.

That's why we built [VDBBench](https://github.com/zilliztech/VectorDBBench): an open-source benchmark designed from the ground up to simulate production reality. Unlike synthetic tests that cherry-pick scenarios, VDBBench pushes databases through continuous ingestion, rigorous filtering conditions, and diverse scenarios, just like your actual production workloads. Our mission is simple: give engineers a tool that shows how vector databases actually perform under real-world conditions so you can make infrastructure decisions based on trustworthy numbers.


## The Gap between Benchmarks and Reality

Traditional benchmarking approaches suffer from three critical flaws that render their results practically meaningless for production decision-making:


### 1. Outdated Data

Many benchmarks still rely on outdated datasets like SIFT or[ GloVe](https://zilliz.com/glossary/glove), which bear little resemblance to today's complex, high-dimensional vector embeddings generated by AI models. Consider this: SIFT contains 128-dimensional vectors, while popular embeddings from OpenAI’s embedding models range from 768 to 3072 dimensions. 


### 2. Vanity Metrics

Many benchmarks focus solely on average latency or peak QPS, which creates a distorted picture. These idealized metrics fail to capture the outliers and inconsistencies that actual users experience in production environments. For example, what good is an impressive QPS number if it requires unbounded computational resources that would bankrupt your organization?


### 3. Oversimplified Scenarios

Most benchmarks test only basic, static workloads—essentially the "Hello World" of vector search. For example, they issue search requests only after the entire dataset is ingested and indexed, ignoring the dynamic reality where users search while new data streams in. This simplistic design overlooks the complex patterns defining real production systems such as concurrent queries, filtered searches, and continuous data ingestion.

Recognizing these flaws, we realized the industry needed a **radical shift in benchmarking philosophy**—one grounded in how AI systems actually behave in the wild. That’s why we built [VDBBench](https://github.com/zilliztech/VectorDBBench).


## From Lab to Production: How VDBBench Bridges the Gap

VDBBench doesn't just iterate on outdated benchmarking philosophies—it rebuilds the concept from first principles with one guiding belief: **a benchmark is only valuable if it predicts actual production behavior**.

We've engineered VDBBench to faithfully replicate real-world conditions across three critical dimensions: data authenticity, workload patterns, and performance measurement.


### Modernizing the Dataset

We've completely overhauled the datasets used for vectorDB benchmarking. Instead of legacy test sets like SIFT and GloVe, VDBBench uses vectors generated from state-of-the-art embedding models that power today's AI applications. 

To ensure relevance, especially for use cases like Retrieval-Augmented Generation (RAG), we selected corpora that reflect real-world enterprise and domain-specific scenarios. These range from general-purpose knowledge bases to vertical applications like biomedical question answering and large-scale web search.

|            |                     |                |                 |
| ---------- | ------------------- | -------------- | --------------- |
| **Corpus** | **Embedding Model** | **Dimensions** | **Size**        |
| Wikipedia  | Cohere V2           | 768            | 1M / 10M        |
| BioASQ     | Cohere V3           | 1024           | 1M / 10M        |
| C4         | OpenAI              | 1536           | 500K / 5M       |
| MSMarco V2 | udever-bloom-1b1    | 1536           | 1M / 10M / 138M |

Table: Datasets used in VDBBench

VDBBench also supports custom datasets, letting you benchmark with your own data generated from your specific embedding models for your specific workloads. After all, no dataset tells a better story than your own production data.


### Production-Focused Metric Design

**VDBBench prioritizes metrics that reflect real-world performance, not just lab results.** We’ve redesigned benchmarking around what actually matters in production environments: reliability under load, tail latency, sustained throughput, and accuracy.

- **P95/P99 latency to measure real user experience**: Average/median latency masks the outliers that frustrate real users. That’s why VDBBench focuses on tail latency like P95/P99, revealing what performance 95% or 99% of your queries will actually achieve.

- **Sustainable throughput under load:** A system that performs well for 5 seconds doesn’t cut it in production. VDBBench gradually increases concurrency to find your database’s maximum sustainable queries per second (`max_qps`)—not the peak number under short, ideal conditions. This shows how well your system holds up over time.

- **Recall balanced with performance:** Speed without accuracy is meaningless. Every performance number in VDBBench is paired with recall, so you know exactly how much relevance you’re trading off for throughput. This enables fair, apples-to-apples comparisons between systems with vastly different internal tradeoffs.


### Test Methodology That Reflects Reality

A key innovation in VDBBench’s design is the **separation of serial and concurrent testing**, which helps capture how systems behave under different types of load. For instance, latency metrics are divided as follows:

- `serial_latency_p99` measures system performance under minimal load, where only one request is processed at a time. This represents the _best-case scenario_ for latency.

- `conc_latency_p99` captures system behavior under _realistic, high-concurrency conditions_, where multiple requests arrive simultaneously.


### Two Benchmark Phases

VDBBench separates testing into two crucial phases: 

1. **Serial Test**

This is a single-process run of 1,000 queries. This phase establishes a baseline for ideal performance and accuracy, reporting both `serial_latency_p99` and recall.

2. **Concurrency Test**

This phase simulates a production environment under sustained load.

- **Realistic client simulation**: Each test process operates independently with its own connection and query set. This avoids shared-state (e.g., cache) interference that could distort results.

- **Synchronized start**: All processes begin simultaneously, ensuring that the measured QPS accurately reflects the claimed concurrency level.

These carefully structured methods ensure that the `max_qps` and `conc_latency_p99` values reported by VDBBench are both **accurate and production-relevant**, providing meaningful insights for production capacity planning and system design.

![](https://assets.zilliz.com/Figure_QPS_and_Latency_of_Milvus_16c64g_standalone_at_Varying_Concurrency_Levels_Cohere_1_M_Test_7f2294e87a.png)

_Figure: QPS and Latency of Milvus-16c64g-standalone at Varying Concurrency Levels (Cohere 1M Test). In this test, Milvus is initially underutilized—up to_ **_concurrency level 20_**_, increasing concurrency improves system utilization and results in higher QPS. Beyond_ **_concurrency 20_**_, the system reaches full load: further increases in concurrency no longer improve throughput, and latency rises due to queuing delays._


## Beyond Searching Static Data: The Real Production Scenarios

To the best of our knowledge, VDBBench is the only benchmark tool that tests vector databases across the complete spectrum of production-critical scenarios, including static collection, filtering, and streaming cases.


### Static Collection

Unlike other benchmarks that rush into testing, VDBBench first ensures each database has fully optimized its indexes—a critical production prerequisite that many benchmarks often neglect. This gives you the complete picture:

- Data ingestion time

- Indexing time (the time used to build an optimized index, which dramatically affects search performance)

- Search performance on fully optimized indexes under both serial and concurrent conditions


### Filtering

Vector search in production rarely happens in isolation. Real applications combine vector similarity with metadata filtering ("find shoes that look like this photo but cost under $100"). This filtered vector search creates unique challenges:

- **Filter Complexity**: More scalar columns and logic conditions increase computational demands

- **Filter Selectiveness**: [Our production experience](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md) reveals this as the hidden performance killer—query speeds can fluctuate by orders of magnitude depending on how selective filters are

VDBBench systematically evaluates filter performance across varying selectivity levels (from 50% to 99.9%), delivering a comprehensive profile of how databases handle this critical production pattern.

![](https://assets.zilliz.com/Figure_QPS_and_Recall_of_Milvus_and_Open_Search_Across_Different_Filter_Selectivity_Levels_Cohere_1_M_Test_4b5df2244d.png)

_Figure: QPS and Recall of Milvus and OpenSearch Across Different Filter Selectivity Levels (Cohere 1M Test). The X-axis represents the percentage of data filtered. As shown, Milvus maintains consistently high recall across all filter selectivity levels, while OpenSearch exhibits unstable performance, with recall fluctuating significantly under different filtering conditions._


### Streaming

Production systems rarely enjoy the luxury of static data. New information continuously flows in while searches execute—a scenario where many otherwise impressive databases collapse.

VDBBench's unique streaming test case examines search-while-inserting performance, measuring:

1. **Impact of Growing Data Volume**: How search performance scales with increasing data size.

2. **Impact of Write Load**: How concurrent writes affect search latency and throughput, as write also consumes CPU or memory resources in the system.

Streaming scenarios represent a comprehensive stress test for any vector database. But building a _fair_ benchmark for this isn’t trivial. It’s not enough to describe how one system behaves—we need a consistent evaluation model that enables **apples-to-apples comparisons** across different databases.

Drawing from our experience helping enterprises with real-world deployments, we built a structured, repeatable approach. With VDBBench:

- You **define a fixed insertion rate** that mirrors your target production workload.

- VDBBench then applies **identical load pressure** across all systems, ensuring performance results are directly comparable.

For example, with a Cohere 10M dataset and a 500 rows/second ingestion target:

- VDBBench spins up 5 parallel producer processes, each inserting 100 rows per second.

- After every 10% of data is ingested, VDBBench triggers a round of search testing under both serial and concurrent conditions.

- Metrics such as latency, QPS, and recall are recorded after each stage.

This controlled methodology reveals how each system's performance evolves over time and under real operational stress—giving you the insight you need to make infrastructure decisions that scale.

![](https://assets.zilliz.com/igure_QPS_and_Recall_of_Pinecone_vs_Elasticsearch_in_the_Cohere_10_M_Streaming_Test_500_rows_s_Ingestion_Rate_548fc02f24.png)

_Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test (500 rows/s Ingestion Rate). Pinecone maintained higher QPS and recall, showing a significant QPS improvement after inserting 100% of the data._

But this is not the end of the story. VDBBench goes even further by supporting an optional optimization step, allowing users to compare streaming search performance before and after index optimization. It also tracks and reports the actual time spent on each stage, offering deeper insights into system efficiency and behavior under production-like conditions.


​![](https://assets.zilliz.com/Figure_QPS_and_Recall_of_Pinecone_vs_Elasticsearch_in_the_Cohere_10_M_Streaming_Test_After_Optimization_500_rows_s_Ingestion_Rate_d249d290bb.png)


_Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test After Optimization (500 rows/s Ingestion Rate)_ 

As shown in the diagram, ElasticSearch surpassed Pinecone in QPS—after index optimization. A miracle? Not quite. The right chart tells the full story: once the x-axis reflects actual elapsed time, it’s clear that ElasticSearch took significantly longer to reach that performance. And in production, that delay matters. This comparison reveals a key tradeoff: peak throughput vs. time-to-serve.


## Choose Your Vector Database with Confidence

The gap between benchmark results and real-world performance shouldn't be a guessing game. VDBBench provides a way to evaluate vector databases under realistic, production-like conditions, including continuous data ingestion, metadata filtering, and streaming workloads.

If you're planning to deploy a vector database in production, it's worth understanding how it performs beyond idealized lab tests. VDBBench is open-source, transparent, and designed to support meaningful, apples-to-apples comparisons.

Try VDBBench with your own workloads today and see how different systems hold up in practice: <https://github.com/zilliztech/VectorDBBench>. 

Have questions or want to share your results? Join the conversation on[ GitHub](https://github.com/zilliztech/VectorDBBench) or connect with our community on [Discord](https://discord.com/invite/FG6hMJStWu). We’d love to hear your opinions. 


Benchmarks Lie — Vector DBs Deserve a Real Test




[Milvus](https://milvus.io/docs/overview.md) is an open-source, highly scalable vector database powering semantic search at a billion-vector scale. As users deploy RAG chatbots, AI customer service, and visual search at this magnitude, a common challenge emerges: **infrastructure costs.** In contrast, exponential business growth is exciting; skyrocketing cloud bills are not. Fast vector search typically requires storing vectors in memory, which is expensive. Naturally, you might ask: _Can we compress vectors to save space without sacrificing search quality?_

The answer is **YES**, and in this blog, we'll show you how implementing a novel technique called [**RaBitQ**](https://dl.acm.org/doi/pdf/10.1145/3654970) enables Milvus to serve 3× more traffic with lower memory cost while maintaining comparable accuracy. We’ll also share the practical lessons learned from integrating RaBitQ into open-source Milvus and the fully-managed Milvus service on [Zilliz Cloud](https://zilliz.com/cloud).

![](https://assets.zilliz.com/Bring_Vector_Compression_to_the_Extreme_How_Milvus_Serves_3_More_Queries_with_Ra_Bit_Q_12f5b4d932.jpg)


## Understanding Vector Search and Compression

Before diving into RaBitQ, let's understand the challenge. 

[**Approximate Nearest Neighbor (ANN)**](https://zilliz.com/glossary/anns) search algorithms are at the heart of a vector database, finding the top-k vectors closest to a given query. A vector is a coordinate in high-dimensional space, often comprising hundreds of floating-point numbers. As vector data scales up, so do storage and compute demands. For instance, running [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW) (an ANN search algorithm) with one billion 768-dimensional vectors in FP32 requires over 3TB of memory!

Like MP3 compresses audio by discarding frequencies imperceptible to the human ear, vector data can be compressed with minimal impact on search accuracy. Research shows that full-precision FP32 is often unnecessary for ANN.[ Scalar Quantization](https://zilliz.com/learn/scalar-quantization-and-product-quantization) (SQ), a popular compression technique, maps floating-point values into discrete bins and stores only the bin indices using low-bit integers. Quantization methods significantly reduce memory usage by representing the same information with fewer bits. Research in this domain strives to achieve the most savings with the least loss in accuracy.

The most extreme compression technique—1-bit Scalar Quantization, also known as [Binary Quantization](https://zilliz.com/learn/scalar-quantization-and-product-quantization)—represents each float with a single bit. Compared to FP32 (32-bit encoding), this reduces memory usage by 32×. Since memory is often the main bottleneck in vector search, such compression can significantly boost performance. **The challenge, however, lies in preserving search accuracy.** Typically, 1-bit SQ reduces recall to below 70%, making it practically unusable.

This is where **RaBitQ** stands out—an excellent compression technique that achieves 1-bit quantization while preserving high recall. Milvus now supports RaBitQ starting from version 2.6, enabling the vector database to serve 3× the QPS while maintaining a comparable level of accuracy.

## A Brief Intro to RaBitQ

[RaBitQ](https://dl.acm.org/doi/pdf/10.1145/3654970) is a smartly designed binary quantization method that leverages the geometry property of high-dimensional space to achieve efficient and accurate vector compression.

At first glance, reducing each dimension of a vector to a single bit may seem too aggressive, but in high-dimensional space, our intuitions often fail us. As Jianyang Gao, an author of RaBitQ,[ illustrated](https://dev.to/gaoj0017/quantization-in-the-counterintuitive-high-dimensional-space-4feg), high-dimensional vectors exhibit the property that individual coordinates tend to be tightly concentrated around zero, a result of a counterintuitive phenomenon explained in[ Concentration of Measure](https://en.wikipedia.org/wiki/Concentration_of_measure). This makes it possible to discard much of the original precision while still preserving the relative structure needed for an accurate nearest neighbor search.

![](https://assets.zilliz.com/Figure_The_counterintuitive_value_distribution_in_high_dimensional_geometry_fad6143bfd.png)

Figure: The counterintuitive value distribution in high-dimensional geometry. _Consider the value of the first dimension for a random unit vector uniformly sampled from the unit sphere; the values are uniformly spread in 3D space. However, for high-dimensional space (e.g., 1000D), the values concentrate around zero, an unintuitive property of high-dimensional geometry. (Image source: [Quantization in The Counterintuitive High-Dimensional Space](https://dev.to/gaoj0017/quantization-in-the-counterintuitive-high-dimensional-space-4feg))_ 

Inspired by this property of high-dimensional space, **RaBitQ focuses on encoding angular information rather than exact spatial coordinates**. It does this by normalizing each data vector relative to a reference point such as the centroid of the dataset. Each vector is then mapped to its nearest vertex on the hypercube, allowing representation with just 1 bit per dimension. This approach naturally extends to `IVF_RABITQ`, where normalization is done relative to the closest cluster centroid, improving local encoding accuracy.

![](https://assets.zilliz.com/Figure_Compressing_a_vector_by_finding_its_closest_approximation_on_the_hypercube_so_that_each_dimension_can_be_represented_with_just_1_bit_cd0d50bb30.png)

_Figure: Compressing a vector by finding its closest approximation on the hypercube, so that each dimension can be represented with just 1 bit. (Image source:_ [_Quantization in The Counterintuitive High-Dimensional Space_](https://dev.to/gaoj0017/quantization-in-the-counterintuitive-high-dimensional-space-4feg)_)_ 

To ensure search remains reliable even with such compressed representations, RaBitQ introduces a **theoretically grounded, unbiased estimator** for the distance between a query vector and binary-quantized document vectors. This helps minimize reconstruction error and sustain high recall.

RaBitQ is also highly compatible with other optimization techniques, such as[ FastScan](https://www.vldb.org/pvldb/vol9/p288-andre.pdf) and[ random rotation preprocessing](https://github.com/facebookresearch/faiss/wiki/Pre--and-post-processing). Moreover, RaBitQ is **lightweight to train and fast to execute**. Training involves simply determining the sign of each vector component, and search is accelerated via fast bitwise operations supported by modern CPUs. Together, these optimizations enable RaBitQ to deliver high-speed search with minimal loss of accuracy.


## Engineering RaBitQ in Milvus: From Academic Research to Production

While RaBitQ is conceptually straightforward and accompanied by a[ reference implementation](https://github.com/gaoj0017/RaBitQ), adapting it in a distributed, production-grade vector database like Milvus presented several engineering challenges. We've implemented RaBitQ in Knowhere, the core vector search engine behind Milvus, and also contributed an optimized version to the open-source ANN search library[ FAISS](https://github.com/facebookresearch/faiss).

Let's look at how we brought this algorithm to life in Milvus.


### Implementation Tradeoffs 

One important design decision involved handling per-vector auxiliary data. RaBitQ requires two floating-point values per vector pre-computed during indexing time, and a third value that can either be computed on-the-fly or pre-computed. In Knowhere, we pre-computed this value at indexing time and stored it to improve efficiency during search. In contrast, the FAISS implementation conserves memory by calculating it at query time, taking a different tradeoff between memory usage and query speed. 

### Hardware Acceleration

Modern CPUs offer specialized instructions that can significantly accelerate binary operations. We tailored the distance computation kernel to take advantage of modern CPU instructions. Since RaBitQ relies on popcount operations, we created a specialized path in Knowhere that uses the `VPOPCNTDQ` instructions for AVX512 when available. On supported hardware (e.g., Intel IceLake or AMD Zen 4), this can accelerate binary distance computations by several factors compared to default implementations.


### Query Optimization

Both Knowhere (Milvus’s search engine) and our optimized FAISS version support scalar quantization (SQ1–SQ8) on query vectors. This provides additional flexibility: even with 4-bit query quantization, recall remains high while computational demands decrease significantly, which is particularly useful when queries must be processed at high throughput. 

We go a step further in optimizing our proprietary Cardinal engine, which powers the fully managed Milvus on Zilliz Cloud. Beyond the capabilities of the open-source Milvus, we introduce advanced enhancements, including integration with a graph-based vector index, additional layers of optimization, and support for Arm SVE instructions.


## The Performance Gain: 3× More QPS with Comparable Accuracy

Starting with version 2.6, Milvus introduces the new `IVF_RABITQ` index type. This new index combines RaBitQ with IVF clustering, random rotation transformation, and optional refinement to deliver an optimal balance of performance, memory efficiency, and accuracy.


### Using IVF_RABITQ in Your Application

Here's how to implement `IVF_RABITQ` in your Milvus application:


```
from pymilvus import MilvusClient

index_params = MilvusClient.prepare_index_params()

index_params.add_index(
    field_name="your_vector_field_name", # Name of the vector field to be indexed
    index_type="IVF_RABITQ", # Will be introduced in Milvus 2.6
    index_name="vector_index", # Name of the index to create
    metric_type="IP", # IVF_RABITQ supports IP and COSINE
    params={
        "nlist": 1024, # IVF param, specifies the number of clusters
    } # Index building params
)
```

### Benchmarking: Numbers Tell the Story

We benchmarked different configurations using[ vdb-bench](https://github.com/zilliztech/vectordbbench), an open-source benchmarking tool for evaluating vector databases. Both the test and control environments use Milvus Standalone deployed on AWS EC2 `m6id.2xlarge` instances. These machines feature 8 vCPUs, 32 GB of RAM, and an Intel Xeon 8375C CPU based on the Ice Lake architecture, which supports the VPOPCNTDQ AVX-512 instruction set.

We used the Search Performance Test from vdb-bench, with a dataset of 1 million vectors, each with 768 dimensions. Since the default segment size in Milvus is 1 GB, and the raw dataset (768 dimensions × 1M vectors × 4 bytes per float) totals roughly 3 GB, the benchmarking involved multiple segments per database.

![](https://assets.zilliz.com/Figure_Example_test_configuration_in_vdb_bench_000142f634.png)

Figure: Example test configuration in vdb-bench.

Here are some low level details about the configuration knobs for IVF, RaBitQ and refinement process: 

- `nlist` and `nprobe` are standard parameters for all `IVF`-based methods

- `nlist` is a non-negative integer that specifies the total number of IVF buckets for the dataset.

- `nprobe` is a non-negative integer that specifies the number of IVF buckets that are visited for a single data vector during the search process. It is a search-related parameter.

- `rbq_bits_query` specifies the level of quantization of a query vector. Use 1..8 values for the `SQ1`..`SQ8` levels of quantization. Use 0 value to disable quantization. It is a search-related parameter.

- `refine`, `refine_type` and `refine_k` parameters are standard parameters for the refine process

- `refine` is a boolean that enables the refinement strategy.

- `refine_k` is a non-negative fp-value. The refining process uses a higher quality quantization method to pick the needed number of nearest neighbors from a `refine_k` times larger pool of candidates, chosen using `IVFRaBitQ`. It is a search-related parameter.

- `refine_type` is a string that specifies the quantization type for a refining index. Available options are `SQ6`, `SQ8`, `FP16`, `BF16` and `FP32` / `FLAT`.

The results reveal important insights:

![](https://assets.zilliz.com/Figure_Cost_and_performance_comparison_of_baseline_IVF_FLAT_IVF_SQ_8_and_IVF_RABITQ_with_different_refinement_strategies_9f69fa449f.png)

Figure: Cost and performance comparison of baseline (IVF_FLAT), IVF_SQ8 and IVF_RABITQ with different refinement strategies

Compared to the baseline `IVF_FLAT` index, which achieves 236 QPS with 95.2% recall, `IVF_RABITQ` reaches significantly higher throughput—648 QPS with FP32 queries and 898 QPS when paired with SQ8-quantized queries. These numbers demonstrate the performance advantage of RaBitQ, especially when refinement is applied.

However, this performance comes with a noticeable trade-off in recall. When `IVF_RABITQ` is used without refinement, recall levels off at around 76%, which may fall short for applications that require high accuracy. That said, achieving this level of recall using 1-bit vector compression is still impressive.

Refinement is essential for recovering accuracy. When configured with SQ8 query and SQ8 refinement, `IVF_RABITQ` delivers both great performance and recall. It maintains a high recall of 94.7%, nearly matching IVF_FLAT, while achieving 864 QPS, over 3× higher than IVF_FLAT. Even compared with another popular quantization index `IVF_SQ8`, `IVF_RABITQ` with SQ8 refinement achieves more than half of the throughput at similar recall, only with a marginal more cost. This makes it an excellent option for scenarios that demand both speed and accuracy.

In short, `IVF_RABITQ` alone is great for maximizing throughput with acceptable recall, and becomes even more powerful when paired with refinement to close the quality gap, using only a fraction of the memory space compared to `IVF_FLAT`.


## Conclusion

RaBitQ marks a significant advancement in vector quantization technology. Combining binary quantization with smart encoding strategies, it achieves what seemed impossible: extreme compression with minimal accuracy loss.

Starting with version 2.6, Milvus will introduce IVF_RABITQ, integrating this powerful compression technique with IVF clustering and refinement strategies to bring binary quantization to production. This combination creates a practical balance between accuracy, speed, and memory efficiency that can transform your vector search workloads.

We're committed to bringing more innovations like this to both open-source Milvus and its fully managed service on Zilliz Cloud, making vector search more efficient and accessible for everyone.


## Getting Started with Milvus 2.6

Milvus 2.6 is available now. In addition to RabitQ, it introduces dozens of new features and performance optimizations such as tiered storage, Meanhash LSH, and enhanced full-text search and multitenancy, directly addressing the most pressing challenges in vector search today: scaling efficiently while keeping costs under control.

Ready to explore everything Milvus 2.6 offers? Dive into our[ release notes](https://milvus.io/docs/release_notes.md), browse the[ complete documentation](https://milvus.io/docs), or check out our[ feature blogs](https://milvus.io/blog). 

If you have any questions or have a similar use case, feel free to reach out to us through our [Discord community](https://discord.com/invite/8uyFbECzPX) or file an issue on[ GitHub](https://github.com/milvus-io/milvus) — we're here to help you make the most of Milvus 2.6.







We Extracted OpenClaw’s Memory System and Open-Sourced It (memsearch)