

You can give Claude Code a 1M-token context window and still get worse answers over time. The issue is not only context size. It is context quality.

Claude Code sessions degrade when terminal logs, raw tool output, repeated file reads, verbose responses, and forgotten project history all compete for attention. In long-running agent workflows, that noise turns into a loop: the model loses the thread, you add more turns to fix the answer, and those extra turns add even more noise.

This is **context defocus**: the model has enough room to hold information, but the important information is buried under low-signal context. Bigger windows can make this easier to ignore because developers stop thinking carefully about what enters the prompt.

![Prompt caching diagram showing how reused prefixes can still add billed context across turns](https://assets.zilliz.com/claude_code_context_management_tools_13_3e7a004cd6.png)

Prompt caching can reduce repeated-prefix cost, but it does not turn the context window into a junk drawer. You still pay for new tokens, and you still need the model to reason over the right information. 

This article reviews seven open-source tools that attack context defocus from different layers: terminal output, tool output, codebase navigation, file reading, model verbosity, semantic code retrieval, and cross-session memory. It also explains how these ideas map to [vector database](https://zilliz.com/learn/what-is-vector-database) design, [vector similarity search](https://zilliz.com/learn/vector-similarity-search), and retrieval systems such as Milvus.

## What causes Claude Code context defocus?

Claude Code context defocus usually comes from five failure modes: too much raw instruction text, noisy tool output, repeated codebase exploration, long model responses, and memory gaps across sessions or agents. 

![Five causes of Claude Code context loss: redundant instructions, messy tool output, repeated codebase retrieval, long responses, and memory gaps](https://assets.zilliz.com/claude_code_context_management_tools_15_56a2da172a.png)

| Context failure mode | What it looks like in Claude Code | Tool category that helps |
| --- | --- | --- |
| Terminal logs are noisy | `git`, `pytest`, `gh`, and cloud CLIs dump more text than the model needs. | CLI output compression |
| Tool outputs flood the window | Test logs, DOM dumps, and MCP outputs enter the chat as giant raw blocks. | Tool-output sandboxing |
| Codebase navigation repeats | Claude lists directories, greps, reads files, and repeats the same exploration every session. | Code graph or semantic retrieval |
| File reads are too broad | The model reads a whole file when it only needed one symbol or summary. | Progressive code reading |
| Claude talks too much | The answer itself adds unnecessary context for future turns. | Response compression |
| Memory does not persist | You re-explain project decisions every time you start a new session. | Markdown-first memory |

A good context-management stack should do three things: keep junk out, retrieve the right project knowledge on demand, and preserve durable decisions across sessions.

## Which Claude Code context tool should you use first?

Start with the layer that creates the most noise in your workflow. If your terminal output is the problem, start with RTK. If Claude keeps wandering through a large repository, start with claude-context or code-review-graph. If your real pain is re-explaining the same decisions every day, start with memsearch.

| Tool | Main problem it solves | Best fit |
| --- | --- | --- |
| [RTK](https://github.com/rtk-ai/rtk) | Noisy terminal output from common developer commands. | Developers who run many CLI commands inside Claude Code. |
| [Context Mode](https://github.com/mksglu/context-mode) | Massive raw tool outputs entering the main conversation. | Heavy Playwright, GitHub, log, or MCP-tool users. |
| [code-review-graph](https://github.com/tirth8205/code-review-graph) | Blind codebase exploration in large repos. | Reviews, dependency analysis, and blast-radius questions. |
| [Token Savior](https://github.com/Mibayy/token-savior) | Full file reads when a symbol summary would be enough. | Large files, repeated symbol lookups, and incremental code reading. |
| [Caveman](https://github.com/JuliusBrussee/caveman) | Claude's own verbose response habits. | Users who want terse output and smaller future context. |
| [claude-context](https://github.com/zilliztech/claude-context) | Re-exploring the codebase every session. | Semantic code search through MCP. |
| [memsearch](https://github.com/zilliztech/memsearch) | Losing project memory across sessions, agents, and model switches. | Long-running projects with durable decisions and lessons. |

The first five tools reduce what enters or remains in context. The last two make useful context easier to recall.

## RTK compresses raw command output before Claude sees it

![](https://assets.zilliz.com/rtk_de6e8e7fb3.png)

RTK is a CLI proxy for reducing token usage from common developer commands. Its GitHub description says it reduces LLM token consumption by 60-90% on common dev commands, and it ships as a single Rust binary.

In everyday Claude Code use, commands like `git status`, `pytest`, and directory listings often dump full environment info and status descriptions into the context window. The model usually needs only a smaller answer: which files changed, which test failed, where the PR is stuck, or what key files exist in the directory.

RTK sits between the shell and Claude. It can rewrite commands through Claude Code hooks and pass back compressed output.

Raw `git status` output:

```bash
On branch feat/payment-retry
Your branch is up to date with 'origin/feat/payment-retry'.

Changes not staged for commit:
  modified:   src/webhook/handler.ts
  modified:   src/queue/dlq.ts
  modified:   tests/webhook.test.ts

Untracked files:
  docs/notes.md

no changes added to commit
```

What actually matters:

```bash
3 modified, 1 untracked
- src/webhook/handler.ts
- src/queue/dlq.ts
- tests/webhook.test.ts
```

Same story with `pytest`. The raw output is full of passing cases and environment noise:

```markdown
============================= test session starts =============================
platform darwin -- Python 3.12.4, pytest-8.4.1
collected 128 items

tests/test_auth.py ....................................
tests/test_webhook.py ....F....
tests/test_queue.py ...................................

================================== FAILURES ==================================
________________ test_retry_to_dlq __________________
E   AssertionError: expected status code 202, got 500
```

Compressed, the signal is immediate:

```apache
128 tests collected, 1 failed
FAIL tests/test_webhook.py::test_retry_to_dlq
AssertionError: expected status code 202, got 500
```

RTK is the easiest starting point when your context bloat comes from shell commands rather than code retrieval.

## Context Mode sandboxes giant tool outputs outside the main chat

Context Mode is built for the raw blocks that tools return: test logs, browser DOM snapshots, GitHub payloads, MCP tool output, and scraped pages. Its GitHub description highlights context-window optimization for AI coding agents and reports 98% tool-output reduction.

![Context Mode GitHub repository card showing sandboxed tool output and context optimization positioning](https://assets.zilliz.com/claude_code_context_management_tools_5_f00e17ea6f.png)

Its approach is to isolate large tool outputs into a local sandbox and index, then pass only summaries and retrieval handles into the Claude conversation.

![Context Mode flow showing large tool output moving through sandbox execution, SQLite or FTS indexes, summaries, and retrieval results](https://assets.zilliz.com/claude_code_context_management_tools_12_32e50fec44.png)

The flow is useful because a coding agent often needs the failing node, broken selector, or relevant stack trace, not the entire DOM or every passing test line. Context Mode keeps the full output available locally while preventing it from dominating the main conversation.

This is similar to how production [hybrid search](https://zilliz.com/blog/hybrid-search-with-milvus) systems separate storage from retrieval. You keep the raw data somewhere durable, then retrieve only the slice that matters. 

## code-review-graph maps code structure before Claude navigates it

code-review-graph addresses a different problem: Claude does not always need more text; it needs a better map.

![code-review-graph logo image used in the original article](https://assets.zilliz.com/claude_code_context_management_tools_8_6d4632f3c9.png)

In a large repository, a simple question can trigger expensive exploration:

> After changing this login logic, which files and tests are affected?

Without a code graph, Claude's typical move is:

```perl
read auth.ts
grep login
read middleware
read tests
keep guessing
```

code-review-graph pre-builds a structural map of the codebase. It uses Tree-sitter to parse functions, classes, imports, call relationships, inheritance, and test dependencies, then writes the graph into SQLite.

That makes it useful for code review and blast-radius analysis. Instead of asking Claude to rediscover the dependency graph through repeated reads, you let it query structure first.

This is adjacent to [semantic search](https://zilliz.com/blog/semantic-search-vs-lexical-search-vs-full-text-search), but not identical. A structural graph answers "what depends on what?" Semantic retrieval answers "what code is conceptually related to this question?" In real code-assistant workflows, you often want both.

## Token Savior gives Claude symbol summaries before full files

Token Savior's core idea is simple: do not send the full file by default. Send an index or symbol summary first, then expand only when the task needs more detail.

![Token Savior GitHub repository card showing its MCP server description and project statistics](https://assets.zilliz.com/claude_code_context_management_tools_10_5f63ad36d1.png)

If you ask where a payment webhook is handled, the model often does not need every line of every related file. It first needs to know whether a file or symbol is relevant.

Token Savior serves code in layers:

| Layer | What Claude receives | When it expands |
| --- | --- | --- |
| Summary | Index, symbol names, and short descriptions. | Default first response. |
| Snippet | A smaller code section around the relevant symbol. | When the summary is likely relevant. |
| Full file | The complete file content. | Only when editing or deep reasoning requires it. |

This mirrors how developers actually read code. You scan, confirm relevance, then open the full file only when necessary. It also resembles the progressive retrieval pattern used in [RAG applications](https://zilliz.com/blog/metadata-filtering-hybrid-search-or-agent-in-rag-applications): retrieve broadly enough to orient, then narrow the context before generation.

## Caveman reduces Claude's own response bloat

Most context tools focus on what enters the model. Caveman targets what Claude outputs.

Caveman is a Claude Code skill/plugin that strips filler, pleasantries, wrapper sentences, over-explanation, and repetitive structures. The goal is not to remove knowledge; it is to make the answer denser.

Without Caveman:

> The reason your React component is re-rendering is likely because...

With Caveman:

> New object ref each render. Inline object prop = new ref = re-render. Wrap in useMemo.

This matters because Claude's own answers become future context. If every answer includes a long explanation, the next turn starts with more text than it needs. Shorter answers can improve the next turn as much as they improve the current one.

For teams thinking about [context engineering for AI agents](https://zilliz.com/blog/context-engineering-for-ai-agents), Caveman is a reminder that output policy is part of context policy.

## claude-context adds semantic code search through MCP

claude-context solves the repeated-codebase-exploration problem with semantic retrieval. It indexes a repository, stores code chunks in a vector database, and exposes search through the [Model Context Protocol](https://zilliz.com/glossary/model-context-protocol-%28mcp%29).

![Claude Context repository shown on GitHub Trending in the original article](https://assets.zilliz.com/claude_code_context_management_tools_2_a0cc547fe3.png)

In a big codebase, you constantly ask Claude questions like:

> Help me figure out which parts of the code might be related to this bug.

Without a retrieval layer, Claude's default approach is often:

```perl
list the directory
grep around
read a bunch of files
keep guessing
```

claude-context moves that work into a retrieval layer. It chunks the repository, generates embeddings, stores them in a [Milvus-backed code index](https://milvus.io/blog/claude-context-reduce-claude-code-token-usage.md), and retrieves relevant code chunks before the model starts reading files blindly.

![claude-context flow showing codebase chunking, embeddings, vector database and hybrid search, relevant code retrieval, and Claude context injection](https://assets.zilliz.com/claude_code_context_management_tools_11_f9d952830d.png)

This is where AI coding tools start to look like search systems. You need chunking, embeddings, metadata, lexical matching, ranking, and freshness. Those are the same building blocks behind [production RAG retrieval](https://zilliz.com/blog/top-10-context-engineering-techniques-you-should-know-for-production-rag), [hybrid retrieval routing](https://milvus.io/blog/build-smarter-rag-routing-hybrid-retrieval.md), and [embedding model selection](https://milvus.io/blog/choose-embedding-model-rag-2026.md).

## memsearch keeps useful memory across sessions and agents

memsearch tackles the opposite side of the problem: not what to forget, but how to recall what matters.

![memsearch logo image from the original article](https://assets.zilliz.com/claude_code_context_management_tools_7_d39c2f200e.png)

Imagine you tell Claude on Monday:

> Our webhook can't retry on failure — failed events need to go into a dead letter queue.

On Wednesday, you open a new session and ask:

> What else can we optimize in the webhook layer?

Without durable memory, Claude treats Monday's decision as if it never happened. You explain it again.

memsearch stores memory as local, human-readable Markdown files and uses Milvus as a rebuildable retrieval index. That design keeps memory editable by humans while still making it searchable for agents.

At retrieval time, memsearch uses progressive recall: search first, expand if needed, then drill down to the original transcript only when necessary.

![memsearch progressive retrieval flow showing search, expand, transcript, and summarized return to the main conversation](https://assets.zilliz.com/20260507_210137_73d4b0b8ea.png)

This Markdown-first pattern is useful for teams working across sessions, models, and agents. It also pairs naturally with [long-term AI agent memory](https://milvus.io/blog/adding-persistent-memory-to-claude-code-with-the-lightweight-memsearch-plugin.md), [shared multi-agent memory](https://milvus.io/blog/openagents-milvus-how-to-build-smarter-multi-agent-systems-that-share-memory.md), and the broader problem of preventing [context rot in agent systems](https://zilliz.com/ai-faq/can-context-engineering-help-reduce-context-rot).

## How do these tools work together?

The seven tools are complementary, not interchangeable. Use them as layers.

| Layer | Use these tools | Why |
| --- | --- | --- |
| Remove command noise | RTK | Compress high-volume terminal output before it reaches Claude. |
| Sandbox raw tool output | Context Mode | Keep large logs, DOMs, and tool payloads outside the main conversation. |
| Map code structure | code-review-graph | Answer dependency and blast-radius questions without blind file reads. |
| Read code progressively | Token Savior | Start with symbol summaries, then expand only as needed. |
| Compress Claude's answers | Caveman | Prevent the model's own output from becoming future context bloat. |
| Retrieve relevant code | claude-context | Use semantic and hybrid code search instead of repeated grep loops. |
| Reuse durable decisions | memsearch | Recall project history across sessions, agents, and model switches. |

A practical rollout order is:

1. **Kill obvious noise first.** Add RTK or Context Mode if shell output and tool payloads dominate your context.
2. **Fix repository navigation.** Add code-review-graph for structure or claude-context for semantic code retrieval.
3. **Control what remains.** Use Token Savior and Caveman to keep file reads and model responses compact.
4. **Preserve durable knowledge.** Use memsearch when repeated explanations become the bottleneck.


## Keep in touch

- Join the [Milvus Discord community](https://discord.com/invite/8uyFbECzPX) to ask questions and compare context-management patterns with other developers.
- [Book a free Milvus Office Hours session](https://milvus.io/office-hours) if you want help designing a retrieval layer for code, memory, or RAG workloads.
- If you'd rather skip the infrastructure setup, [Zilliz Cloud](https://cloud.zilliz.com/signup) (managed Milvus) offers a free tier to get started.

## Frequently Asked Questions

**How do I reduce Claude Code token usage without losing useful context?**

Start by compressing the noisiest inputs: terminal output, raw tool payloads, and repeated code reads. Then add retrieval tools such as claude-context or code-review-graph so Claude can pull relevant code instead of exploring the repository from scratch.

**Should I use claude-context or code-review-graph for a large repo?**

Use claude-context when you need semantic code search, especially when you do not know the exact file or symbol name. Use code-review-graph when you need structural answers such as call relationships, imports, test dependencies, and review blast radius.

**Is memory different from code retrieval in Claude Code?**

Yes. Code retrieval finds relevant project files or symbols. Memory retrieval recalls durable decisions, user preferences, debugging history, and cross-session lessons. memsearch focuses on memory; claude-context focuses on code retrieval.

**Do these tools replace prompt caching or a larger context window?**

No. Prompt caching and large context windows help with capacity and cost, but they do not decide what information deserves attention. Context-management tools improve the quality and density of what enters the model in the first place.
![cccm 11zon](https://assets.zilliz.com/cccm_11zon_848f7f1c6b.png)


7 Best Open-Source Tools for Claude Code Context Management


Claude Context: Cut Claude Code Token Usage with Milvus


Claude Context: Reduce Claude Code Token Usage with Milvus-Powered Code Retrieval



*This post is contributed by Song Zhi, CEO of Senqi AI, an embodied-AI company building task-execution infrastructure for robots. RoboBrain is one of Senqi AI's core products.*

Most robot capabilities work fine on their own. A navigation model can plan a route. A perception model can identify objects. A speech module can accept instructions. The production failure appears when those capabilities have to run as one continuous task.

For a robot, a simple instruction like "go check that area, photograph anything unusual, and notify me" requires planning before the task starts, adapting while it runs, and producing a useful result when it finishes. Each handoff can break: navigation freezes behind an obstacle, a blurry photo is accepted as final, or the system forgets the exception it handled five minutes ago.

That is the core challenge for [AI agents](https://zilliz.com/glossary/ai-agents) operating in the physical world. Unlike digital agents, robots execute against continuous [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data): blocked paths, changing light, battery limits, sensor noise, and operator rules.

RoboBrain is Senqi AI's embodied-intelligence operating system for robot task execution. It sits at the task layer, connecting perception, planning, execution control, and data feedback so natural-language instructions can become structured, recoverable robot workflows.

| Breakpoint | What Fails in Production | How RoboBrain Closes It |
|---|---|---|
| Task planning | Vague instructions leave downstream modules without concrete execution fields. | Task objectification turns intent into shared state. |
| Context routing | The right information exists, but reaches the wrong decision stage. | Tiered memory routes real-time, short-term, and long-term context separately. |
| Data feedback | A single pass completes or fails without improving the next run. | Feedback writebacks update task state and long-term memory. |

## Three Breakpoints in Robot Task Execution

Software tasks can often be bounded as input, process, and result. Robot tasks run against a moving physical state: blocked paths, changing light, battery limits, sensor noise, and operator rules.

That is why the task loop needs more than isolated models. It needs a way to preserve context across planning, execution, and feedback.

### 1. Task Planning: Vague Instructions Produce Vague Execution

A phrase like "go take a look" hides a lot of decisions. Which area? What should the robot photograph? What counts as unusual? What should it do if the shot fails? What result should it return to the operator?

If the task layer cannot resolve those details into concrete fields — target area, inspection object, completion condition, failure policy, and return format — the task runs without direction from the start and never recovers context downstream.

### 2. Context Routing: The Right Data Reaches the Wrong Stage

The robot stack may already contain the right information, but task execution depends on retrieving it at the right stage.

The startup phase needs maps, area definitions, and operating rules. Mid-execution needs live sensor state. Exception handling needs similar cases from prior deployments. When those sources are mixed up, the system makes the right kind of decision with the wrong context.

When routing fails, startup pulls stale experience instead of area rules, exception handling cannot reach the cases it needs, and mid-execution gets yesterday's map instead of live readings. Giving someone a dictionary does not help them write an essay. The data has to reach the right decision point, at the right stage, in the right form.

### 3. Data Feedback: Single-Pass Execution Does Not Improve

Without writeback, a robot can finish a run without improving the next one. A completed action still needs a quality check: is the image sharp enough, or should the robot reshoot? Is the path still clear, or should it detour? Is the battery above threshold, or should the task terminate?

A single-pass system has no mechanism for those calls. It executes, stops, and repeats the same failure next time.

## How RoboBrain Closes the Robot Task Loop

RoboBrain connects environment understanding, task planning, execution control, and data feedback into one operating loop.

![RoboBrain core middleware architecture showing how user intent flows through task objects, stage-aware memory powered by Milvus, and a policy engine before reaching embodied capabilities](https://assets.zilliz.com/how_robobrain_builds_longterm_robot_memory_with_milvus_2_660d1c90e3.png)

In the architecture described in this contributed post, that loop is implemented through three mechanisms:

1. **Task objectification** structures the entry point.
2. **Tiered memory** routes the right information to the right stage.
3. **A feedback loop** writes results back and decides the next move.

They only work as a set. Fix one without the others and the chain still breaks at the next point.

### 1. Task Objectification: Turning Intent into Shared State

Before execution starts, RoboBrain turns each instruction into a task object: task type, target area, inspection object, constraints, expected output, current stage, and failure policy.

The point is not just parsing language. The point is giving every downstream module the same stateful view of the task. Without that conversion, the task has no direction.

For the patrol example, the task object fills in the inspection type, designated zone, anomalous items as the check object, battery >= 20% as the constraint, a clear anomaly photo plus operator alert as the expected output, and return-to-base as the failure policy.

The stage field updates as the run changes. An obstacle moves the task from navigating to detouring or requesting help. A blurry image moves it from inspecting to reshooting. A low battery moves it to termination and return-to-base.

Downstream modules no longer receive isolated commands. They receive the current task stage, its constraints, and the reason the stage changed.

### 2. Tiered Memory: Routing Context to the Right Stage

RoboBrain splits task-relevant information into three tiers so the right data reaches the right stage.

**Real-time state** holds pose, battery, sensor readings, and environmental observations. It supports decisions at every control step.

**Short-term context** records events within the current task: the obstacle the robot avoided two minutes ago, the photo it reshot, or the door it failed to open on the first try. It keeps the system from losing track of what just happened.

**Long-term semantic memory** stores scene knowledge, historical experience, exception cases, and post-task writebacks. A particular parking area may require camera-angle adjustments at night because of reflective surfaces. A certain anomaly type may have a history of false positives and should trigger human review instead of an automatic alert.

This long-term tier runs on [vector similarity search](https://zilliz.com/learn/vector-similarity-search) through the [Milvus vector database](https://milvus.io/), because retrieving the right memory means matching by meaning, not by ID or keyword. Scene descriptions and handling records are stored as [vector embeddings](https://zilliz.com/glossary/vector-embeddings) and retrieved with [approximate nearest neighbor search](https://zilliz.com/glossary/anns) to find the closest semantic matches.

Startup pulls area rules and past patrol summaries from long-term memory. Mid-execution relies on real-time state and short-term context. Exception handling uses [semantic search](https://zilliz.com/glossary/semantic-search) to find similar cases in long-term memory.

### 3. Feedback Loop: Writing Results Back into the System

RoboBrain writes navigation, perception, and action results back to the task object after each step, updating the stage field. The system reads those observations and decides the next move: detour if the path is unreachable, reshoot if the image is blurry, retry if the door will not open, or terminate if battery is low.

Execution becomes a cycle: execute, observe, adjust, execute again. The chain keeps adapting to environmental changes instead of cutting off the first time something unexpected shows up.

## How Milvus Powers RoboBrain's Long-Term Robot Memory

Some robot memory can be queried by task ID, timestamp, or session metadata. Long-term operational experience usually cannot.

The useful record is often the case that is semantically similar to the current scene, even if the task ID, location name, or wording is different. That makes it a [vector database](https://zilliz.com/learn/what-is-vector-database) problem, and Milvus is a fit for the long-term memory tier.

This tier stores information such as:

- Area rule descriptions and point-location semantics
- Anomaly type definitions and example summaries
- Historical handling records and post-task review conclusions
- Patrol summaries written at task completion
- Experience writebacks after human takeover
- Failure causes and correction strategies from similar scenarios

None of that is naturally keyed by a structured field. All of it needs to be recalled by meaning.

A concrete example: the robot patrols a parking lot entrance at night. Glare from overhead lights makes anomaly detection unstable. Reflections keep getting flagged as anomalies.

The system needs to recall reshoot strategies that worked under strong nighttime glare, camera-angle corrections from similar areas, and human-review conclusions that marked earlier detections as false positives. An exact-match query can find a known task ID or time window. It cannot reliably surface "the prior glare case that behaved like this one" unless that relationship has already been labeled.

Semantic similarity is the retrieval pattern that works. [Similarity metrics](https://zilliz.com/blog/similarity-metrics-for-vector-search) rank stored memories by relevance, while [metadata filtering](https://milvus.io/docs/filtered-search.md) can narrow the search space by area, task type, or time window. In practice, this often becomes [hybrid search](https://zilliz.com/learn/hybrid-search-combining-text-and-image): semantic matching for meaning, structured filters for operational constraints.

For implementation, the filter layer is often where semantic memory becomes operational. [Milvus filter expressions](https://milvus.io/docs/boolean.md) define scalar constraints, while [Milvus scalar queries](https://milvus.io/docs/get-and-scalar-query.md?file=query.md) support exact lookups when the system needs records by metadata rather than similarity.

This retrieval pattern resembles [retrieval-augmented generation](https://zilliz.com/learn/Retrieval-Augmented-Generation) adapted for physical-world decision-making rather than text generation. The robot is not retrieving documents to answer a question; it is retrieving prior experience to choose the next safe action.

Not everything goes into Milvus. Task IDs, timestamps, and session metadata live in a relational database. Raw runtime logs live in a logging system. Each storage system handles the query pattern it is built for.

| Data Type | Where It Lives | How It Is Queried |
|---|---|---|
| Task IDs, timestamps, session metadata | Relational database | Exact lookups, joins |
| Raw runtime logs and event streams | Logging system | Full-text search, time-range filters |
| Scene rules, handling cases, experience writebacks | Milvus | Vector similarity search by meaning |

As tasks run and scenes accumulate, the long-term memory layer feeds downstream processes: sample curation for model fine-tuning, broader data analysis, and cross-deployment knowledge transfer. The memory compounds into a data asset that gives every future deployment a higher starting point.

## What This Architecture Changes in Deployment

Task objectification, tiered memory, and the feedback loop turn RoboBrain's task loop into a deployment pattern: each task preserves state, each exception can retrieve prior experience, and each run can improve the next one.

A robot patrolling a new building should not start from scratch if it has already handled similar lighting, obstacles, anomaly types, or operator rules elsewhere. That is what makes robot task execution more repeatable across scenes, and what makes long-term deployment costs easier to control.

For robotics teams, the deeper lesson is that memory is not just a storage layer. It is part of execution control. The system needs to know what it is doing, what just changed, what similar cases have happened before, and what should be written back for the next run.

## Further Reading

If you are working on similar problems with robot memory, task execution, or semantic retrieval for embodied AI, these resources are useful next steps:

- Read the [Milvus documentation](https://milvus.io/docs) or try the [Milvus quickstart](https://milvus.io/docs/quickstart.md) to see how vector search works in practice.
- Review the [Milvus architecture overview](https://milvus.io/docs/architecture_overview.md) if you are planning a production memory layer.
- Browse [vector database use cases](https://zilliz.com/vector-database-use-cases) for more examples of semantic search in production systems.
- Join the [Milvus community](https://milvus.io/community) to ask questions and share what you are building.
- If you want managed Milvus instead of running your own infrastructure, learn more about [Zilliz Cloud](https://zilliz.com/cloud).

How RoboBrain Builds Long-Term Robot Memory with Milvus


DeepSeek V4 RAG Benchmark with Milvus vs GPT-5.5 and Qwen


DeepSeek V4 vs GPT-5.5 vs Qwen3.6: Which Model Should You Use?



[**Hermes Agent**](https://github.com/NousResearch/hermes-agent) **has been everywhere lately.** Built by Nous Research, Hermes is a self-hosted personal AI agent that runs on your own hardware (a \$5 VPS works) and talks to you through existing chat channels like Telegram.

**Its biggest highlight is a built-in learning loop:** the loop creates Skills from experience, improves them during use, and searches past conversations to find reusable patterns. Other agent frameworks hand-code Skills before deployment. Hermes's Skills grow from use, and repeated workflows become reusable with zero code change.

**The catch is that Hermes's retrieval is keyword-only.** It matches exact words, but not the meaning users are after. When users use different wording across different sessions, the loop can't connect them, and no new Skill gets written. When there are only a few hundred documents, the gap is tolerable. **Past that, the loop stops learning because it can't find its own history.**

**The fix is Milvus 2.6.** Its [hybrid search](https://milvus.io/docs/multi-vector-search.md) covers both meaning and exact keywords in a single query, so the loop can finally connect rephrased information across sessions. It's light enough to fit on a small cloud server (a \$5/month VPS runs it). Swapping it in doesn't require changing Hermes — Milvus slots behind the retrieval layer, so the Learning Loop stays intact. Hermes still picks which Skill to run, and Milvus handles what to retrieve.

But the deeper payoff goes beyond better recall: once retrieval works, the Learning Loop can store the retrieval strategy itself as a Skill -- not just the content it retrieves. That's how the agent's knowledge work compounds across sessions.

## Hermes Agent Architecture: How Four-Layer Memory Powers the Skill Learning Loop

[**Hermes**](https://github.com/NousResearch/hermes-agent) **has four memory layers, and L4 Skills is the one that sets it apart.**

-   **L1** — session context, cleared when the session closes
-   **L2** — persisted facts: project stack, team conventions, resolved decisions
-   **L3** — SQLite FTS5 keyword search over local files
-   **L4** — stores workflows as Markdown files. Unlike LangChain tools or AutoGPT plugins, which developers author in code before deployment, L4 Skills are self-written: they grow from what the agent actually runs, with zero developer authoring.

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_3_3653368e99.png)

## Why Hermes's FTS5 Keyword Retrieval Breaks the Learning Loop

**Hermes needs retrieval to trigger cross-session workflows in the first place.** But its built-in L3 layer uses SQLite FTS5, which only matches literal tokens, not meaning.

**When users phrase the same intent differently across sessions, FTS5 misses the match.** The Learning Loop doesn't fire. No new Skill gets written, and next time the intent comes around, the user is back to routing by hand.

Example: the knowledge base stores "asyncio event loop, async task scheduling, non-blocking I/O." A user searches "Python concurrency." FTS5 returns zero hits — no literal word overlap, and FTS5 has no way to see that they're the same question.

Under a couple hundred documents, the gap is tolerable. Past that, documentation uses one vocabulary, and users ask in another, and FTS5 has no bridge between them. **Unretrievable content might as well not be in the knowledge base, and the Learning Loop has nothing to learn from.**

## How Milvus 2.6 Fixes the Retrieval Gap with Hybrid Search and Tiered Storage

**Milvus 2.6 brings two upgrades that fit Hermes's failure points.** **Hybrid search** unblocks the Learning Loop by covering both semantic and keyword retrieval in one call. **Tiered storage** keeps the whole retrieval backend small enough to run on the same \$5/month VPS Hermes was built for.

### What Hybrid Search Solves: Finding Relevant Information

Milvus 2.6 supports running both vector retrieval (semantic) and [BM25 full-text search](https://milvus.io/docs/full-text-search.md) (keyword) in a single query, then merging the two ranked lists with [Reciprocal Rank Fusion (RRF)](https://milvus.io/docs/multi-vector-search.md).

For example: ask "what is the principle of asyncio", and vector retrieval hits semantically related content. Ask "where is the `find_similar_task` function defined", and BM25 precisely matches the function name in code. For questions that involve a function inside a particular type of task, hybrid search returns the right result in one call, with no hand-written routing logic.

For Hermes, this is what unblocks the Learning Loop. When a second session rephrases the intent, vector retrieval catches the semantic match FTS5 missed. The loop fires, and a new Skill gets written.

### What Tiered Storage Solves: Cost

A naive vector database would want the full embedding index in RAM, which pushes personal deployments toward bigger, more expensive infrastructure. Milvus 2.6 avoids that with three-tier storage, moving entries between tiers based on access frequency:

-   **Hot** — in memory
-   **Warm** — on SSD
-   **Cold** — on object storage

Only hot data stays resident. A 500-document knowledge base fits under 2 GB of RAM. The whole retrieval stack runs on the same \$5/month VPS Hermes targets, with no infrastructure upgrade needed.

## Hermes + Milvus: System Architecture

**Hermes picks which Skill to run. Milvus handles what to retrieve.** The two systems stay separate, and Hermes's interface doesn't change.

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_4_1794304940.png)

**The flow:**

1.  Hermes identifies the user's intent and routes to a Skill.
2.  The Skill calls a retrieval script through the terminal tool.
3.  The script hits Milvus, runs hybrid search, and returns ranked chunks with source metadata.
4.  Hermes composes the answer. Memory records the workflow.
5.  When the same pattern repeats across sessions, the Learning Loop writes a new Skill.

## How to Install Hermes and Milvus 2.6

**Install Hermes and** [**Milvus 2.6 Standalone**](https://milvus.io/docs/install_standalone-docker.md)**, then create a collection with dense and BM25 fields.** That's the full setup before the Learning Loop can fire.

### Install Hermes

```bash
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
```

Run `hermes` to enter the interactive init wizard:

-   **LLM provider** — OpenAI, Anthropic, OpenRouter (OpenRouter has free models)
-   **Channel** — this walkthrough uses a FLark bot

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_5_dceeae1519.png)

### Run Milvus 2.6 Standalone

Single-node standalone is enough for a personal agent:

```bash
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh \
-o standalone_embed.sh
bash standalone_embed.sh start
# Verify service status
docker ps | grep milvus
```

### Create the Collection

Schema design caps what retrieval can do. This schema runs dense vectors and BM25 sparse vectors side by side:

```python
from pymilvus import MilvusClient, DataType, Function, FunctionType
client = MilvusClient(
    uri="http://192.168.x.x:19530",
)
schema = client.create_schema(auto_id=True, enable_dynamic_field=True)
schema.add_field("id", DataType.INT64, is_primary=True)
# Raw text (for BM25 full-text search)
schema.add_field(
    "text",
    DataType.VARCHAR,
    max_length=8192,
    enable_analyzer=True,
    enable_match=True
)
# Dense vector (semantic search)
schema.add_field("dense_vector", DataType.FLOAT_VECTOR, dim=1536)
# Sparse vector (BM25 auto-generated, Milvus 2.6 feature)
schema.add_field("sparse_vector", DataType.SPARSE_FLOAT_VECTOR)
schema.add_field("source", DataType.VARCHAR, max_length=512)
schema.add_field("chunk_index", DataType.INT32)
# Tell Milvus to auto-convert text to sparse_vector via BM25
bm25_function = Function(
    name="text_bm25",
    function_type=FunctionType.BM25,
    input_field_names=["text"],
    output_field_names=["sparse_vector"],
)
schema.add_function(bm25_function)
index_params = client.prepare_index_params()
# HNSW graph index (dense vector)
index_params.add_index(
    field_name="dense_vector",
    index_type="HNSW",
    metric_type="COSINE",
    params={"M": 16, "efConstruction": 256}
)
# BM25 inverted index (sparse vector)
index_params.add_index(
    field_name="sparse_vector",
    index_type="SPARSE_INVERTED_INDEX",
    metric_type="BM25"
)
client.create_collection(
    collection_name="hermes_milvus",
    schema=schema,
    index_params=index_params
)
```

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_6_0646f46d36.png)

### Hybrid Search Script

```python
import sys, json
from openai import OpenAI
from pymilvus import MilvusClient, AnnSearchRequest, RRFRanker

client = MilvusClient("http://192.168.x.x:19530")
oai    = OpenAI()
COLLECTION = "hermes_milvus"

def hybrid_search(query: str, top_k: int = 5) -> list[dict]:
    # 1. Vectorize query
    dense_vec = oai.embeddings.create(
        model="text-embedding-3-small",
        input=query
    ).data[0].embedding

    # 2. Dense vector retrieval (semantic relevance)
    dense_req = AnnSearchRequest(
        data=[dense_vec],
        anns_field="dense_vector",
        param={"metric_type": "COSINE", "params": {"ef": 128}},
        limit=top_k * 2       # Widen candidate set, let RRF do final ranking
    )

    # 3. BM25 sparse vector retrieval (exact term matching)
    bm25_req = AnnSearchRequest(
        data=[query],
        anns_field="sparse_vector",
        param={"metric_type": "BM25"},
        limit=top_k * 2
    )

    # 4. RRF fusion ranking
    results = client.hybrid_search(
        collection_name=COLLECTION,
        reqs=[dense_req, bm25_req],
        ranker=RRFRanker(k=60),
        limit=top_k,
        output_fields=["text", "source", "doc_type"]
    )

    return [
        {
            "text":     r.entity.get("text"),
            "source":   r.entity.get("source"),
            "doc_type": r.entity.get("doc_type"),
            "score":    round(r.distance, 4)
        }
        for r in results[0]
    ]

if __name__ == "__main__":
    query= sys.argv[1] if len(sys.argv) > 1 else ""
    top_k  = int(sys.argv[2]) if len(sys.argv) > 2 else 5
    output = hybrid_search(query, top_k)
    print(json.dumps(output, ensure_ascii=False, indent=2))
```

**The dense request widens the candidate pool by 2× so RRF has enough to rank from.** `text-embedding-3-small` is the cheapest OpenAI embedding that still holds retrieval quality; swap in `text-embedding-3-large` if the budget allows.

With the environment and knowledge base ready, the next section puts the Learning Loop to the test.

## Hermes Skill Auto-Generation in Practice

**Two sessions show the Learning Loop in action.** In the first, the user names the script by hand. In the second, a new session asks the same question without naming the script. Hermes picks up the pattern and writes three Skills.

### Session 1: Call the Script by Hand

Open Hermes in Lark. Give it the script path and the retrieval target. Hermes invokes the terminal tool, runs the script, and returns the answer with source attribution. **No Skill exists yet. This is a plain tool call.**

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_7_1c2d9261f4.png)

### Session 2: Ask Without Naming the Script

Clear the conversation. Start fresh. Ask the same category of question without mentioning the script or path.

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_8_27253eda82.png)

### Memory Writes First, Skill Follows

**The Learning Loop records the workflow (script, arguments, return shape) and returns the answer.** Memory holds the trace; no Skill exists yet.

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_9_a0768f84bd.png)

**The second session's match tells the loop the pattern is worth keeping.** When it fires, three Skills get written:

| Skill | Role |
| --- | --- |
| `hybrid-search-doc-qa` | Run hybrid semantic + keyword search over Memory and compose the answer |
| `milvus-docs-ingest-verification` | Verify documents have been ingested into the knowledge base |
| `terminal` | Run shell commands: scripts, environment setup, inspection |

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_10_b68e35bc46.png)

From this point on, **users stop naming Skills.** Hermes infers intent, routes to the Skill, pulls the relevant chunks from Memory, and writes the answer. There's no Skill selector in the prompt.

Most RAG (retrieval-augmented generation) systems solve the storing-and-fetching problem, but the fetch logic itself is hard-coded in application code. Ask in a different way or in a new scenario, and retrieval breaks. Hermes stores the fetch strategy as a Skill, which means **the fetch path becomes a document you can read, edit, and version.** The line `💾 Memory updated · Skill 'hybrid-search-doc-qa' created` isn't a setup-complete marker. It's **the Agent committing a behavior pattern to long-term memory.**

## Hermes vs. OpenClaw: Accumulation vs. Orchestration

**Hermes and OpenClaw answer different problems.** Hermes is built for a single agent that accumulates memory and skills across sessions. OpenClaw is built for breaking a complex task into pieces and handing each piece to a specialized agent.

![](https://assets.zilliz.com/hermes_agent_learning_loop_milvus_hybrid_search_md_11_afcb575d50.png)

OpenClaw's strength is orchestration. It optimizes for how much of a task gets done automatically. Hermes's strength is accumulation: a single agent that remembers across sessions, with skills that grow from use. Hermes optimizes for long-term context and domain experience.

**The two frameworks stack.** Hermes ships a one-step migration path that pulls `~/.openclaw` memory and skills into Hermes's memory layers. An orchestration stack can sit on top, with an accumulation agent underneath.

For the OpenClaw side of the split, see [What Is OpenClaw? Complete Guide to the Open-Source AI Agent](https://milvus.io/blog/openclaw-formerly-clawdbot-moltbot-explained-a-complete-guide-to-the-autonomous-ai-agent.md) on the Milvus blog.

## Conclusion

Hermes's Learning Loop turns repeated workflows into reusable Skills, but only if retrieval can connect them across sessions. FTS5 keyword search can't. [**Milvus 2.6 hybrid search**](https://milvus.io/docs/multi-vector-search.md) can: dense vectors handle meaning, BM25 handles exact keywords, RRF merges both, and [tiered storage](https://milvus.io/docs/tiered-storage-overview.md) keeps the whole stack on a \$5/month VPS.

The bigger point: once retrieval works, the agent doesn't just store better answers: it stores better retrieval strategies as Skills. The fetch path becomes a versionable document that improves with use. That's what separates an agent that accumulates domain expertise from one that starts fresh every session. For a comparison of how other agents handle (or fail to handle) this problem, see [Claude Code's Memory System Explained.](https://milvus.io/blog/claude-code-memory-memsearch.md)

## Get Started

**Try the tools in this article:**

-   [Hermes Agent on GitHub](https://github.com/NousResearch/hermes-agent) — install script, provider setup, and channel configuration used above.
-   [Milvus 2.6 Standalone Quickstart](https://milvus.io/docs/install_standalone-docker.md) — single-node Docker deploy for the knowledge-base backend.
-   [Milvus Hybrid Search Tutorial](https://milvus.io/docs/multi-vector-search.md) — full dense + BM25 + RRF example matching the script in this post.

**Got questions about Hermes + Milvus hybrid search?**

-   Join the [Milvus Discord](https://discord.gg/milvus) to ask about hybrid search, tiered storage, or Skill-routing patterns — other developers are building similar stacks.
-   [Book a Milvus Office Hours session](https://milvus.io/community#office-hours) to walk through your own agent + knowledge-base setup with the Milvus team.

**Want to skip the self-host?**

-   [Sign up](https://cloud.zilliz.com/signup) or [sign in](https://cloud.zilliz.com/login) to Zilliz Cloud — managed Milvus with hybrid search and tiered storage out of the box. New work-email accounts get **\$100 in free credits**.

## Further Reading

-   [Milvus 2.6 release notes](https://milvus.io/docs/release_notes.md) — tiered storage, hybrid search, schema changes
-   [Zilliz Cloud & Milvus CLI + Official Skills](https://zilliz.com/blog) — operational tooling for Milvus-native agents
-   [Why RAG-Style Knowledge Management Breaks for Agents](https://zilliz.com/blog) — the case for agent-specific memory design
-   [Claude Code's Memory System Is More Primitive Than You'd Expect](https://zilliz.com/blog) — comparison piece on another agent's memory stack

## Frequently Asked Questions

### How does Hermes Agent's Skill Learning Loop actually work?

Hermes records every workflow it runs -- the script called, arguments passed, and return shape -- as a memory trace. When the same pattern appears across two or more sessions, the Learning Loop fires and writes a reusable Skill: a Markdown file that captures the workflow as a repeatable procedure. From that point on, Hermes routes to the Skill by intent alone, without the user naming it. The critical dependency is retrieval -- the loop only fires if it can find the earlier session's trace, which is why keyword-only search becomes a bottleneck at scale.

### What's the difference between hybrid search and vector-only search for agent memory?

Vector-only search handles meaning well but misses exact matches. If a developer pastes an error string like ConnectionResetError or a function name like find_similar_task, a pure vector search might return semantically related but wrong results. Hybrid search combines dense vectors (semantic) with BM25 (keyword) and merges the two result sets with Reciprocal Rank Fusion. For agent memory -- where queries range from vague intent ("Python concurrency") to exact symbols -- hybrid search covers both ends in a single call without routing logic in your application layer.

### Can I use Milvus hybrid search with AI agents other than Hermes?

Yes. The integration pattern is generic: the agent calls a retrieval script, the script queries Milvus, and results return as ranked chunks with source metadata. Any agent framework that supports tool calls or shell execution can use the same approach. Hermes happens to be a strong fit because its Learning Loop specifically depends on cross-session retrieval to fire, but the Milvus side is agent-agnostic -- it doesn't know or care which agent is calling it.

### How much does a self-hosted Milvus + Hermes setup cost per month?

A single-node Milvus 2.6 Standalone on a 2-core / 4 GB VPS with tiered storage runs about \$5/month. OpenAI text-embedding-3-small costs \$0.02 per 1M tokens -- a few cents per month for a personal knowledge base. LLM inference dominates total cost and scales with usage, not with the retrieval stack.

How to Fix Hermes Agent's Learning Loop with Milvus 2.6 Hybrid Search



Anthropic's [Managed Agents](https://www.anthropic.com/engineering/managed-agents) make agent infrastructure resilient. A 200-step task now survives a harness crash, a sandbox timeout, or a mid-flight infrastructure change without human intervention, and Anthropic reports p50 time-to-first-token dropped roughly 60% and p95 dropped over 90% after the decoupling.

What reliability doesn't solve is memory. A 200-step code migration that hits a new dependency conflict on step 201 can't efficiently look back at how it handled the last one. An agent running vulnerability scans for one customer has no idea that another agent already solved the same case an hour ago. Every session starts on a blank page, and parallel brains have no access to what the others have already worked out.

The fix is to pair the [Milvus vector database](https://milvus.io/) with Anthropic's Managed Agents: semantic recall within a session, and a shared [vector memory layer](https://milvus.io/docs/milvus_for_agents.md) across sessions. The session contract stays untouched, the harness gets one new layer, and long-horizon agent tasks get qualitatively different capabilities.

## What Managed Agents Solved (and What They Didn't)

**Managed Agents solved reliability by decoupling the agent into three independent modules. What it didn't solve is memory, either as semantic recall inside a single session or as shared experience across parallel sessions.** Here's what got decoupled, and where the memory gap sits inside that decoupled design.

| Module | What it does |
| --- | --- |
| **Session** | An append-only event log of everything that happened. Stored outside the harness. |
| **Harness** | The loop that calls Claude and routes Claude's tool calls to the relevant infrastructure. |
| **Sandbox** | The isolated execution environment where Claude runs code and edits files. |

The reframe that makes this design work is stated explicitly in Anthropic's post:

_"The session is not Claude's context window."_

The context window is ephemeral: bounded in tokens, reconstructed per model call, and discarded when the call returns. The session is durable, stored outside the harness, and represents the system of record for the entire task.

![](https://assets.zilliz.com/anthropic_managed_agents_memory_milvus_md_3_edae1b022d.png)

When a harness crashes, the platform starts a fresh one with `wake(sessionId)`. The new harness reads the event log via `getSession(id)`, and the task picks up from the last recorded step, with no custom recovery logic to write and no session-level babysitting to operate.

What the Managed Agents post doesn't address, and doesn't claim to, is what the agent does when it needs to remember anything. Two gaps show up the moment you push real workloads through the architecture. One lives inside a single session; the other lives across sessions.

## Problem 1: Why Linear Session Logs Fail Past a Few Hundred Steps

**Linear session logs fail past a few hundred steps because sequential reads and semantic search are fundamentally different workloads, and the** `**getEvents()**` **API serves only the first one.** Slicing by position or seeking to a timestamp is enough to answer "where did this session leave off." It is not enough to answer the question an agent will predictably need on any long task: have we seen this kind of problem before, and what did we do about it?

Consider a code migration at step 200 that hits a new dependency conflict. The natural move is to look back. Did the agent run into something similar earlier in this same task? What approach was tried? Did it hold, or did it regress something else downstream?

With `getEvents()` there are two ways to answer that, and both are bad:

| Option | Problem |
| --- | --- |
| Scan every event sequentially | Slow at 200 steps. Untenable at 2,000. |
| Dump a large slice of the stream into the context window | Expensive on tokens, unreliable at scale, and crowds out the agent's actual working memory for the current step. |

![](https://assets.zilliz.com/anthropic_managed_agents_memory_milvus_md_4_2ac29b32af.png)

The session is good for recovery and audit, but it was not built with an index that supports "have I seen this before." Long-horizon tasks are where that question stops being optional.

## Solution 1: How to Add Semantic Memory to a Managed Agent's Session

**Add a Milvus collection alongside the session log and dual-write from** `**emitEvent**`**.** The session contract stays untouched, and the harness gains semantic query over its own past.

![](https://assets.zilliz.com/anthropic_managed_agents_memory_milvus_md_5_404a1048aa.png)

Anthropic's design leaves headroom for exactly this. Their post states that "any fetched events can also be transformed in the harness before being passed to Claude's context window. These transformations can be whatever the harness encodes, including context organization... and context engineering." Context engineering lives in the harness; the session only has to guarantee durability and queryability.

The pattern: every time `emitEvent` fires, the harness also computes a [vector embedding](https://zilliz.com/learn/everything-you-need-to-know-about-vector-embeddings) for events worth indexing and inserts them into a Milvus collection.

```python
from pymilvus import MilvusClient

milvus_client = MilvusClient(uri="http://localhost:19530")

# Only index high-signal events. Tool retries and intermediate states are noise.
INDEXABLE_EVENT_TYPES = {"decision", "strategy", "resolution", "error_handling"}

async def emit_event(session_id: str, event: dict):
    # Original path: append to the session event stream.
    await session_store.append(session_id, event)

    # Extended path: embed the event content and insert into Milvus.
    if event["type"] in INDEXABLE_EVENT_TYPES:
        embedding = await embed(event["content"])
        milvus_client.insert(
            collection_name="agent_memory",
            data=[{
                "vector":     embedding,
                "session_id": session_id,
                "step":       event["step"],
                "event_type": event["type"],
                "content":    event["content"],
            }]
        )
```

When the agent hits step 200 and needs to recall prior decisions, the query is a [vector search](https://zilliz.com/glossary/vector-similarity-search) scoped to that session:

```python
async def recall_similar(query: str, session_id: str, top_k: int = 5):
    query_vector = await embed(query)
    results = milvus_client.search(
        collection_name="agent_memory",
        data=[query_vector],
        filter=f'session_id == "{session_id}"',
        limit=top_k,
        output_fields=["step", "event_type", "content"]
    )
    return results[0]  # top_k most relevant past events
```

Three production details matter before this ships:

-   **Pick what to index.** Not every event deserves an embedding. Tool-call intermediate states, retry logs, and repetitive status events pollute retrieval quality faster than they improve it. The `INDEXABLE_EVENT_TYPES` policy is task-dependent, not global.
-   **Define the consistency boundary.** If the harness crashes between the session append and the Milvus insert, one layer is briefly ahead of the other. The window is small but real. Pick a reconciliation path (retry on restart, write-ahead log, or eventual reconciliation) rather than hoping.
-   **Control embedding spend.** A 200-step session that calls an external embedding API synchronously on every step produces an invoice nobody planned for. Queue embeddings and send them asynchronously in batches.

With those in place, recall takes milliseconds for the vector search plus under 100ms for the embedding call. The top-five most relevant past events land in context before the agent notices friction. The session keeps its original job as the durable log; the harness gains the ability to query its own past semantically rather than sequentially. That's a modest change at the API surface and a structural change in what the agent can do on long-horizon tasks.

## Problem 2: Why Parallel Claude Agents Can't Share Experience

**Parallel Claude agents can't share experience because Managed Agents sessions are isolated by design. The same isolation that makes horizontal scaling clean also prevents every brain from learning from every other brain.**

In a decoupled harness, brains are stateless and independent. That isolation unlocks the latency wins Anthropic reports, and it also keeps every session running in the dark about every other session.

Agent A spends 40 minutes diagnosing a tricky SQL injection vector for one customer. An hour later, Agent B picks up the same case for a different customer and spends its own 40 minutes walking the same dead ends, running the same tool calls, and arriving at the same answer.

For a single user running the occasional agent, that is wasted compute. For a platform running dozens of concurrent [AI agents](https://zilliz.com/glossary/ai-agents) across code review, vulnerability scans, and documentation generation for different customers every day, the cost compounds structurally.

If the experience every session produces evaporates the moment the session ends, the intelligence is disposable. A platform built this way scales linearly but doesn't get better at anything over time, the way human engineers do.

## Solution 2: How to Build a Shared Agent Memory Pool with Milvus

**Build one vector collection that every harness reads from at startup and writes to at shutdown, partitioned by tenant so experience pools across sessions without leaking across customers.**

When a session ends, the key decisions, problems encountered, and approaches that worked are pushed into the shared Milvus collection. When a new brain initializes, the harness runs a semantic query as part of setup and injects the top-matching past experiences into the context window. Step one of the new agent inherits the lessons of every prior agent.

Two engineering decisions carry this from prototype to production.

### Isolating Tenants with the Milvus Partition Key

**Partition by** `**tenant_id**`**, and Customer A's agent experiences physically don't live in the same partition as Customer B's. That's isolation at the data layer rather than a query convention.**

Brain A's work on Company A's codebase should never be retrievable by Company B's agents. Milvus's [partition key](https://milvus.io/docs/use-partition-key.md) handles this on a single collection, with no second collection per tenant and no sharding logic in application code.

```python
# Declare partition key at schema creation.
schema.add_field(
    field_name="tenant_id",
    datatype=DataType.VARCHAR,
    max_length=64,
    is_partition_key=True   # Automatic per-tenant partitioning.
)

# Every query filters by tenant. Isolation is automatic.
results = milvus_client.search(
    collection_name="shared_agent_memory",
    data=[query_vector],
    filter=f'tenant_id == "{current_tenant}"',
    limit=5,
    output_fields=["content", "step", "session_id"]
)
```

Customer A's agent experiences never surface in Customer B's queries, not because the query filter is written correctly (though it has to be), but because the data physically does not live in the same partition as Customer B's. One collection to operate, logical isolation enforced at the query layer, physical isolation enforced at the partition layer.

See the [multi-tenancy strategies docs](https://milvus.io/docs/multi_tenancy.md) for when partition key fits versus when separate collections or databases do, and the [multi-tenancy RAG patterns guide](https://milvus.io/blog/build-multi-tenancy-rag-with-milvus-best-practices-part-one.md) for production deployment notes.

### Why Agent Memory Quality Needs Ongoing Work

**Memory quality erodes over time: flawed workarounds that happened to succeed once get replayed and reinforced, and stale entries tied to deprecated dependencies keep misleading agents that inherit them. The defenses are operational programs, not database features.**

An agent stumbles on a flawed workaround that happens to succeed once. It gets written to the shared pool. The next agent retrieves it, replays it, and reinforces the bad pattern with a second "successful" usage record.

Stale entries follow a slower version of the same path. A fix pinned to a dependency version that was deprecated six months ago keeps getting retrieved, and keeps misleading agents that inherit it. The older and more heavily used the pool, the more of this accumulates.

![](https://assets.zilliz.com/anthropic_managed_agents_memory_milvus_md_6_24f71b1c21.png)

Three operational programs defend against this:

-   **Confidence score.** Track how often a memory has been successfully applied in downstream sessions. Decay entries that fail in replay. Promote entries that succeed repeatedly.
-   **Time weighting.** Prefer recent experiences. Retire entries past a known staleness threshold, often tied to major dependency version bumps.
-   **Human spot checks.** Entries with high retrieval frequency are high-leverage. When one of them is wrong, it is wrong many times, which is where human review pays back fastest.

Milvus alone doesn't solve this, and neither does Mem0, Zep, or any other memory product. Enforcing one pool with many tenants and zero cross-tenant leakage is something you engineer once. Keeping that pool accurate, fresh, and useful is continuous operational work that no database ships pre-configured.

## Takeaways: What Milvus Adds to Anthropic's Managed Agents

**Milvus turns Managed Agents from a reliable-but-forgetful platform into one that compounds experience over time by adding semantic recall inside a session and shared memory across agents.**

Managed Agents answered the reliability question cleanly: both brains and hands are cattle, and any one can die without taking the task with it. That's the infrastructure problem, and Anthropic solved it well.

What stayed open was growth. Human engineers compound over time; years of work turn into pattern recognition, and they don't reason from first principles on every task. Today's managed agents don't, because every session starts on a blank page.

Wiring the session to Milvus for semantic recall inside a task and pooling experience across brains in a shared vector collection is what gives agents a past they can actually use. Plugging in Milvus is the infrastructure piece; pruning wrong memories, retiring stale ones, and enforcing tenant boundaries is the operational piece. Once both are in place, the shape of memory stops being a liability and starts being compounding capital.

## Get Started

-   **Try it locally:** spin up an embedded Milvus instance with [Milvus Lite](https://milvus.io/docs/milvus_lite.md). No Docker, no cluster, just `pip install pymilvus`. Production workloads graduate to [Milvus Standalone or Distributed](https://milvus.io/docs/install_standalone-docker.md) when you need them.
-   **Read the design rationale:** Anthropic's [Managed Agents engineering post](https://www.anthropic.com/engineering/managed-agents) walks through the session, harness, and sandbox decoupling in depth.
-   **Got questions?** Join the [Milvus Discord](https://discord.com/invite/8uyFbECzPX) community for agent memory design discussions, or book a [Milvus Office Hours](https://milvus.io/office-hours) session to walk through your workload.
-   **Prefer managed?** [Sign up for Zilliz Cloud](https://cloud.zilliz.com/signup) (or [sign in](https://cloud.zilliz.com/login)) for hosted Milvus with partition keys, scaling, and multi-tenancy built in. New accounts get free credits on a work email.

## Frequently Asked Questions

**Q: What's the difference between a session and a context window in Anthropic's Managed Agents?**

The context window is the ephemeral set of tokens a single Claude call sees. It's bounded and resets per model invocation. The session is the durable, append-only event log of everything that happened across the whole task, stored outside the harness. When a harness crashes, `wake(sessionId)` spawns a new harness that reads the session log and resumes. The session is the system of record; the context window is working memory. The session is not the context window.

**Q: How do I persist agent memory across Claude sessions?**

The session itself is already persistent; that's what `getSession(id)` retrieves. What's typically missing is queryable long-term memory. The pattern is to embed high-signal events (decisions, resolutions, strategies) into a vector database like Milvus during `emitEvent`, then query by semantic similarity at retrieval time. This gives you both the durable session log Anthropic provides and a semantic recall layer for looking back across hundreds of steps.

**Q: Can multiple Claude agents share memory?**

Not out of the box. Each Managed Agents session is isolated by design, which is what lets them scale horizontally. To share memory across agents, add a shared vector collection (for example in Milvus) that each harness reads from at startup and writes to at shutdown. Use Milvus's partition key feature to isolate tenants so Customer A's agent memories never leak into Customer B's sessions.

**Q: What's the best vector database for AI agent memory?**

The honest answer depends on scale and deployment shape. For prototypes and small workloads, a local embedded option like Milvus Lite runs in-process with no infrastructure. For production agents across many tenants, you want a database with mature multi-tenancy (partition keys, filtered search), hybrid search (vector + scalar + keyword), and millisecond-latency at millions of vectors. Milvus is purpose-built for vector workloads at that scale, which is why it appears in production agent memory systems built on LangChain, Google ADK, Deep Agents, and OpenAgents.

Add Long-Term Memory to Anthropic's Managed Agents with Milvus


How to Add Long-Term Memory to Anthropic's Managed Agents with Milvus



Google's [TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) paper claimed **6x compression, 8x speedup, and near-zero accuracy loss** for vector representations. After it was released, memory and storage stocks fell sharply, and major tech outlets quickly turned it into a headline story.

The market reaction was only the start. Researchers soon began asking whether the paper's claims were overstated and whether it treated prior work — especially [RaBitQ](https://dl.acm.org/doi/10.1145/3654970) — fairly. The dispute pushed **vector quantization** back into the spotlight, partly because the same underlying ideas now matter in two critical parts of the AI stack: [vector search systems](https://zilliz.com/learn/vector-similarity-search) and KV-cache compression for large models.

To understand both the technical debate and what it means for production systems, we spoke with **Cheng Long**, Associate Professor at NTU Singapore and head of VectorDB@NTU; **Jianyang Gao**, first author of RaBitQ; and **Li Liu**, Director of Engineering at Zilliz. The conversation covered vector quantization itself, the questions raised around TurboQuant, and why this matters for systems like [Milvus](https://milvus.io/), the most popular open-source [vector databases](https://zilliz.com/learn/what-is-vector-database), and large-scale vector retrieval.

**_Related reading:_** _If you want the engineering side rather than the interview, see our companion article on_ [_how vector quantization affects AI infrastructure costs_](https://milvus.io/blog/turboquant-rabitq-vector-database-cost.md)_._

## Why did vector quantization suddenly become such a big topic?

**Zilliz: Before we get into the controversy, could you start by explaining what vector quantization is and why it has become so important in AI?**

**Cheng Long:** Vector quantization is a technique for **data compression** and **approximate representation**. It originally came from signal processing, where it was used for image and audio compression. In modern AI systems, its role has changed because vectors have become one of the basic units of computation.

Today, its importance is clearest in two places.

One is **real-time search over collections with billions or even tens of billions of vectors**. In semantic retrieval systems, the core task is similarity search over high-dimensional vectors. But raw vectors are large, and floating-point computation is expensive. At scale, that makes it difficult to deliver millisecond-level latency. Vector quantization helps by compressing vectors into low-bit representations and speeding up distance computation. That is why it matters for practical workloads such as [single-vector search](https://milvus.io/docs/single-vector-search.md), [multi-vector search](https://milvus.io/docs/multi-vector-search.md), and index design in [Milvus search architecture](https://milvus.io/docs/index-explained.md).

The other is **KV cache compression** for large models. KV cache reduces redundant computation during generation, but the memory cost grows quickly as context gets longer. So the problem becomes how to compress those vectors without hurting output quality too much. At its core, that is also a vector quantization problem.

**Zilliz: If vector quantization becomes more widely used — and if TurboQuant's results hold up — does that mean storage demand drops sharply?**

**Jianyang Gao:** Under the same model and the same workload, compression can reduce storage demand. But that does not justify the broader conclusion people jumped to.

When TurboQuant talks about **6x compression** and **8x speedup**, it is comparing against a basic **16-bit/32-bit baseline**. That is not the same as comparing against other methods in the same category. So the real effect still needs to be evaluated more carefully.

**Zilliz: Then from that perspective, if the market reaction were really about the technology itself, should it have happened much earlier, when similar ideas had already appeared?**

**Cheng Long:** From a technical point of view, you could say that similar theoretical territory had already been reached before. But markets do not move in sync with research. There is usually a lag between academic results, engineering adoption, and financial interpretation.

And over a longer horizon, the effect may not even be linear. Compression can make it possible to run large models on smaller devices, which can create new demand rather than simply reduce it. The relationship between technology and markets is more complicated than a straight-line extrapolation.

## How did RaBitQ emerge, and what did it contribute?

**Zilliz: How did you first arrive at the idea for RaBitQ?**

**Jianyang Gao:** We started from a gap we saw in vector databases. Traditional methods such as [Product Quantization](https://milvus.io/docs/ivf-pq.md) worked well empirically, but they offered very little in the way of theoretical guarantees.

At the time, I was studying high-dimensional probability at NTU Singapore, and that led me to ask whether we could build a method that was not only practical, but also came with a clear theoretical guarantee. That was the starting point for RaBitQ.

**Zilliz: What do you see as RaBitQ's core originality?**

**Jianyang Gao:** Its key idea was to use a random rotation, a.k.a., Johnson-Lindenstrauss transformation, to make the distribution of vector coordinates more uniform and more predictable.

Once you have that, you can derive an optimal quantization estimator on top of it. We then gave a strict proof that it reaches the theoretical lower bound.

Earlier work had also tried to introduce random rotation. But from our perspective, those methods did not achieve the effect we were looking for because of practical issues in algorithm design.

**Zilliz: From an engineering perspective, what stood out most to you about RaBitQ?**

**Li Liu:** We had worked with many quantization algorithms, from [scalar quantization methods](https://milvus.io/docs/ivf-sq8.md) to PQ and other variants. What stood out about RaBitQ was that it changed how people approached the problem.

Before that, much of the field was still fairly empirical. You could say a method seemed to work, but it was harder to explain clearly why. RaBitQ approached the problem in a much more mathematical way. The method felt elegant and, in a sense, simple. That way of thinking influenced a lot of later work.

**Zilliz: Put simply, how much can it save in memory and cost?**

**Li Liu:** At the same recall level, moving from 4-bit compression to 2-bit compression cuts memory use by half.

And it is not just about compression. Its performance compares favorably with earlier approaches, and that matters in production environments where teams care about both memory efficiency and retrieval quality. That is why it matters for systems that need to balance [dense vector storage](https://milvus.io/docs/dense-vector.md), throughput, and recall.

**Zilliz: Beyond Milvus, where do you see RaBitQ being used today?**

**Cheng Long:** First, I want to thank the Milvus team, because they were among the earliest to adopt RaBitQ. We also had a lot of discussions and some collaborative research along the way.

RaBitQ has also been adopted in some other systems including Meta’s FAISS, VSAG, VectorChord, Volcengine OpenSearch, CockroachDB, ElasticSearch, Lucene, and turbopuffer. What stands out on the Milvus side is that the team shipped [IVF_RABITQ](https://milvus.io/docs/ivf-rabitq.md) as a real index option in [Milvus 2.6](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md), alongside broader work on [collection management](https://milvus.io/docs/manage-collections.md), [IVF-based indexing](https://milvus.io/docs/ivf-flat.md), and [HNSW-based indexing](https://milvus.io/docs/hnsw.md).

## How should we evaluate TurboQuant?

**Zilliz: In your public response, you said TurboQuant had some serious issues. What, in your view, were the main ones?**

**Jianyang Gao:** We see three main problems.

One is the way the paper describes prior work and discusses overlap. The TurboQuant paper misrepresents the methodology of RaBitQ, ignoring the most similar part, such as Johnson-Lindenstrauss Transformation. Another is the way the paper characterizes the theoretical result. It describes RaBitQ as suboptimal without providing any explanation or evidence, but RaBitQ is optimal in fact. The third is the fairness of the experimental comparison. They use single-core CPU to evaluate RaBitQ while using A100 GPU to evaluate TurboQuant.

**Zilliz: Let's take the benchmark issue first. Why do you think the comparison was not fair?**

**Jianyang Gao:** Benchmark claims only mean something if the setup is comparable. If one system is tested under a very different hardware or software environment, then the result may reflect the setup more than the algorithm itself.

In our view, differences in processor choice, implementation language, and optimization level can make a major difference. That is why benchmark methodology needs to be interpreted very carefully, especially by teams building production retrieval systems.

**Cheng Long:** The paper also made some other claims that do not hold.

For example, the paper says that **RaBitQ cannot be vectorized**. But RaBitQ had already open-sourced code with SIMD-based vectorized computation when the 2024 paper was published. So from our perspective, that statement was factually incorrect.

It is also worth mentioning that we began working with NVIDIA last year and completed a GPU implementation of RaBitQ. The related code is under review for inclusion in NVIDIA's cuVS library.

**Zilliz: Milvus evaluated TurboQuant in the second half of 2025 but did not adopt it. What did your team see in testing?**

**Li Liu:** It does contain one useful idea. In our view, it makes a small optimization in how the quantization grid is allocated. But the most important step in the method — using random rotation for quantization — was first introduced by RaBitQ.

And when it comes to unbiased estimation, RaBitQ's approach is cleaner and its theoretical derivation is stronger.

That said, because this was a result from Google, we tested it in 2025. In our lab, under a standardized CPU environment, TurboQuant did not outperform our internal RaBitQ version in most of the cases we evaluated. So when the market reacted so strongly, we were genuinely surprised.

**Zilliz: For readers who have not looked closely at either paper, could you walk through where RaBitQ and TurboQuant overlap in plain language?**

**Li Liu:** At a high level, both methods begin with **random rotation**. Mathematically, that means multiplying the vector by a random orthogonal matrix. You can think of it as changing your viewing angle in a high-dimensional space. It does not change the relative positions of the data points, but it distributes the information across dimensions more evenly.

After that comes **quantization**. You divide the continuous real-valued space into **2^k grid cells**, where **k** is the number of quantization bits, and then map each vector element to a nearby grid point. TurboQuant makes a small adjustment here by allocating the grid according to the data distribution instead of distributing it evenly.

The last step is **error estimation**, and this is where RaBitQ's main contribution lies. Traditional methods calculate directly from the quantized values, which makes the error harder to control. RaBitQ estimates the quantization error more precisely, and that is where its mathematical optimality comes from. TurboQuant's solution is more complicated, and in our setting the tradeoff did not look as attractive.

## Why is attribution so hard to resolve in practice?

**Zilliz:** After you published your public statement, how did Google and ICLR respond?

**Cheng Long:** ICLR did not take action. We emailed them during the review period in September last year, but did not receive a response. We wrote again in March this year and were told to post comments on OpenReview, but beyond that there was no action.

As for Google, one of the co-authors replied a few days ago. The reply said they would revise the arXiv version to correct its inaccurate description of RaBitQ's optimality.

**Zilliz:** Earlier the discussion was framed around academic misconduct. Now it also sounds like a question of imbalance and who gets to shape the story. Why is it so hard to defend your work?

**Cheng Long:** One problem is scale. AI conferences are now so large that a single cycle can bring in tens of thousands of papers. Organizers simply do not have the capacity to handle every dispute of this kind.

The other problem is imbalance. Large companies have a much stronger public voice. Independent researchers or smaller teams do not have the same communication power.

**Jianyang Gao:** For individuals, the cost is extremely high. Professor Long and I have barely been able to work normally in recent weeks.

The process itself has also been frustrating. We were firmly rejected when we contacted the authors, and we received no response from the conference organizers. In practice, many researchers look at situations like this and decide to let them go. But that is also how many original contributions disappear from the public narrative.

**Zilliz:** It sounds like this is not the first time your team has run into this kind of problem.

**Cheng Long:** No, it is not.

We have seen cases before where companies take RaBitQ, make a few engineering modifications, give it a new name, and then describe it only as something inspired by RaBitQ.

That is why I appreciate the way some industry teams handle this, including Milvus. When they use RaBitQ, they describe it objectively. And when they add optimizations beyond the original version, they explain those clearly as their own engineering contribution. That gives proper credit to the original work while also showing the company's technical strength.

**Zilliz:** When large companies build on academic work, do they usually provide any financial sharing or benefit allocation?

**Jianyang Gao:** In most cases, no.

That said, large companies still have a strong incentive to present a technical advance as something they created themselves rather than something they adopted from others. Everyone wants customers and investors to see the most advanced work as the result of their own team's innovation.

## What comes next for vector quantization?

**Zilliz:** What research directions are you working on now?

**Cheng Long:** A large part of our work will remain focused on vector retrieval.

One direction is to combine RaBitQ with different vector retrieval indexes, such as IVF and HNSW, so the system can support larger-scale data with lower latency, higher concurrency, and lower cost. I am also paying attention to KV cache compression.

**Jianyang Gao:** KV cache in large models and vector retrieval share many of the same properties, both mathematically and at the systems level, because both deal with high-dimensional vectors.

Going forward, I want to think more about how to apply mathematical tools, including ideas from high-dimensional probability, to accelerate inference and training.

**Zilliz:** Where is the ceiling for vector quantization as a field? How much room is left for improvement?

**Cheng Long:** From a theoretical point of view, the ceiling is largely in sight. RaBitQ is already asymptotically optimal.

But there is still a great deal of room on the engineering side. You still have to deal with hardware characteristics, data distribution, latency constraints, and many other practical factors. That is exactly why production systems still need careful work in areas such as [distributed vector database architecture](https://milvus.io/docs/architecture_overview.md), [sparse vector support](https://milvus.io/docs/sparse_vector.md), [reranking pipelines](https://milvus.io/docs/reranking.md), and metric selection in [Milvus distance metrics](https://milvus.io/docs/metric.md).

## Keep Reading

If you want to dig deeper into the engineering side of RaBitQ and how it fits into Milvus, these are the most relevant resources:

-   [IVF_RABITQ documentation](https://milvus.io/docs/ivf-rabitq.md) — configuration details and tuning guidance.
-   [RaBitQ integration deep dive](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3×-more-queries-with-rabitq.md) — how Milvus turned RaBitQ into a production index.
-   [How vector quantization affects AI infrastructure costs](https://milvus.io/blog/turboquant-rabitq-vector-database-cost.md) — our broader analysis of the TurboQuant-RaBitQ discussion.
-   [Milvus 2.6 release post](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md) — where IVF_RABITQ shipped as a real Milvus index option.
-   [Milvus index explained](https://milvus.io/docs/index-explained.md) — how IVF_RABITQ fits with other index choices.
-   [IVF_FLAT indexing](https://milvus.io/docs/ivf-flat.md) and [HNSW indexing](https://milvus.io/docs/hnsw.md) — useful baselines if you are comparing index tradeoffs.
-   [Schema design in Milvus](https://milvus.io/docs/schema.md) and [filtered search](https://milvus.io/docs/filtered-search.md) — useful if you are evaluating RaBitQ in a real application rather than in isolation.
-   [Milvus quickstart](https://milvus.io/docs/quickstart.md) and [RAG system design](https://zilliz.com/learn/Retrieval-Augmented-Generation) — helpful if you want to try this in a retrieval pipeline.

Join the [Milvus Slack community](https://slack.milvus.io/) or [book Milvus Office Hours](https://milvus.io/office-hours) if you want to talk through your workload.

If you'd rather skip infrastructure setup, you can [sign up for Zilliz Cloud](https://cloud.zilliz.com/signup) (fully managed Milvus) .

RaBitQ Authors on the TurboQuant Vector Quantization Dispute


Interview with RaBitQ Authors: The TurboQuant Dispute and Why the Storage Selloff Was a False Alarm


Graph RAG Without a Graph Database | Vector Graph RAG


We Built Graph RAG Without the Graph Database



Mitchell Hashimoto built HashiCorp and co-created Terraform. In February 2026, he published a [blog post](https://mitchellh.com/writing/my-ai-adoption-journey) describing a habit he'd developed while working with AI agents: every time an agent made a mistake, he engineered a permanent fix into the agent's environment. He called it "engineering the harness." Within weeks, [OpenAI](https://openai.com/index/harness-engineering/) and [Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps) published engineering articles expanding on the idea. The term *Harness Engineering* had arrived.

It resonated because it names a problem every engineer building [AI agents](https://zilliz.com/glossary/ai-agents) has already hit. [Prompt engineering](https://zilliz.com/glossary/prompt-as-code-(prompt-engineering)) gets you better single-turn outputs. Context engineering manages what the model sees. But neither addresses what happens when an agent runs autonomously for hours, making hundreds of decisions without supervision. That's the gap Harness Engineering fills — and it almost always depends on hybrid search (hybrid full-text and semantic search) to work.

## What Is Harness Engineering?

Harness Engineering is the discipline of designing the execution environment around an autonomous AI agent. It defines which tools the agent can call, where it gets information, how it validates its own decisions, and when it should stop.

To understand why it matters, consider three layers of AI agent development:

| Layer | What It Optimizes | Scope | Example |
|-------|------------------|-------|---------|
| **Prompt Engineering** | What you say to the model | Single exchange | Few-shot examples, chain-of-thought prompts |
| **Context Engineering** | What the model can see | [Context window](https://zilliz.com/glossary/context-window) | Document retrieval, history compression |
| **Harness Engineering** | The world the agent operates in | Multi-hour autonomous execution | Tools, validation logic, architectural constraints |

**Prompt Engineering** optimizes the quality of a single exchange — phrasing, structure, examples. One conversation, one output.

**Context Engineering** manages how much information the model can see at once — which documents to retrieve, how to compress history, what fits in the context window and what gets dropped.

**Harness Engineering** builds the world the agent operates in. Tools, knowledge sources, validation logic, architectural constraints — everything that determines whether an agent can run reliably across hundreds of decisions without human supervision.

![Three layers of AI agent development: Prompt Engineering optimizes what you say, Context Engineering manages what the model sees, and Harness Engineering designs the execution environment](https://assets.zilliz.com/harness_engineering_ai_agents_4_2f4bc35890.png)

The first two layers shape the quality of a single turn. The third shapes whether an agent can operate for hours without you watching.

These aren't competing approaches. They're a progression. As agent capability grows, the same team moves through all three — often within a single project.

## How OpenAI Used Harness Engineering to Build a Million-Line Codebase and Lessons They Learnt

OpenAI ran an internal experiment that puts Harness Engineering in concrete terms. They described it in their engineering blog post, ["Harness Engineering: Leveraging Codex in an Agent-First World"](https://openai.com/index/harness-engineering/). A three-person team started with an empty repository in late August 2025. For five months, they wrote no code themselves — every line was generated by Codex, OpenAI's AI-powered coding agent. The result: one million lines of production code and 1,500 merged pull requests.

The interesting part isn't the output. It's the four problems they hit and the harness-layer solutions they built.

### Problem 1: No Shared Understanding of the Codebase

What abstraction layer should the agent use? What are the naming conventions? Where did last week's architecture discussion land? Without answers, the agent guessed — and guessed wrong — repeatedly.

The first instinct was a single `AGENTS.md` file containing every convention, rule, and historical decision. It failed for four reasons. Context is scarce, and a bloated instruction file crowded out the actual task. When everything is marked important, nothing is. Documentation rots — rules from week two become wrong by week eight. And a flat document can't be mechanically verified.

The fix: shrink `AGENTS.md` to 100 lines. Not rules — a map. It points to a structured `docs/` directory containing design decisions, execution plans, product specs, and reference docs. Linters and CI verify that cross-links stay intact. The agent navigates to exactly what it needs.

The underlying principle: if something isn't in context at runtime, it doesn't exist for the agent.

### Problem 2: Human QA Couldn't Keep Pace with Agent Output

The team plugged Chrome DevTools Protocol into Codex. The agent could screenshot UI paths, observe runtime events, and query logs with LogQL and metrics with PromQL. They set a concrete threshold: a service had to start in under 800 milliseconds before a task was considered complete. Codex tasks ran for over six hours at a stretch — typically while engineers slept.

### Problem 3: Architectural Drift Without Constraints

Without guardrails, the agent reproduced whatever patterns it found in the repo — including bad ones.

The fix: strict layered architecture with a single enforced dependency direction — Types → Config → Repo → Service → Runtime → UI. Custom linters enforced these rules mechanically, with error messages that included the fix instruction inline.

![Strict layered architecture with one-way dependency validation: Types at the base, UI at the top, custom linters enforce rules with inline fix suggestions](https://assets.zilliz.com/harness_engineering_ai_agents_3_f0fc3c9e92.png)

In a human team, this constraint usually arrives when a company scales to hundreds of engineers. For a coding agent, it's a prerequisite from day one. The faster an agent moves without constraints, the worse the architectural drift.

### Problem 4: Silent Technical Debt

The solution: encode the project's core principles into the repository, then run background Codex tasks on a schedule to scan for deviations and submit refactoring PRs. Most merged automatically within a minute — small continuous payments rather than periodic reckoning.

## Why AI Agents Can't Grade Their Own Work

OpenAI's experiment proved Harness Engineering works. But separate research exposed a failure mode inside it: agents are systematically bad at evaluating their own output.

The problem appears in two forms.

**Context anxiety.** As the context window fills, agents begin wrapping up tasks prematurely — not because the work is done, but because they sense the window limit approaching. Cognition, the team behind the AI coding agent Devin, [documented this behavior](https://cognition.ai/blog/devin-sonnet-4-5-lessons-and-challenges) while rebuilding Devin for Claude Sonnet 4.5: the model became aware of its own context window and started taking shortcuts well before actually running out of room.

Their fix was pure harness engineering. They enabled the 1M-token context beta but capped actual usage at 200K tokens — tricking the model into believing it had ample runway. The anxiety vanished. No model change required; just a smarter environment.

The most common general mitigation is compaction: summarize history and let the same agent continue with compressed context. This preserves continuity but doesn't eliminate the underlying behavior. An alternative is context reset: clear the window, spin up a fresh instance, and hand off state through a structured artifact. This removes the anxiety trigger entirely but demands a complete handoff document — gaps in the artifact mean gaps in the new agent's understanding.

**Self-evaluation bias.** When agents assess their own output, they score it high. Even on tasks with objective pass/fail criteria, the agent spots a problem, talks itself into believing it's not serious, and approves work that should fail.

The fix borrows from GANs (Generative Adversarial Networks): separate the generator from the evaluator completely. In a GAN, two neural networks compete — one generates, one judges — and that adversarial tension forces quality up. The same dynamic applies to [multi-agent systems](https://milvus.io/blog/openagents-milvus-how-to-build-smarter-multi-agent-systems-that-share-memory.md).

Anthropic tested this with a three-agent harness — Planner, Generator, Evaluator — against a solo agent on the task of building a 2D retro game engine. They describe the full experiment in ["Harness Design for Long-Running Application Development"](https://www.anthropic.com/engineering/harness-design-long-running-apps) (Anthropic, 2026). The Planner expands a short prompt into a full product spec, deliberately leaving implementation details unspecified — early over-specification cascades into downstream errors. The Generator implements features in sprints, but before writing code, it signs a sprint contract with the Evaluator: a shared definition of "done." The Evaluator uses Playwright (Microsoft's open-source browser automation framework) to click through the application like a real user, testing UI, API, and database behavior. If anything fails, the sprint fails.

The solo agent produced a game that technically launched, but entity-to-runtime connections were broken at the code level — discoverable only by reading the source. The three-agent harness produced a playable game with AI-assisted level generation, sprite animation, and sound effects.

![Comparison of solo agent versus three-agent harness: solo agent ran 20 minutes at nine dollars with broken core functionality, while the full harness ran 6 hours at two hundred dollars producing a fully functional game with AI-assisted features](https://assets.zilliz.com/harness_engineering_ai_agents_1_38a13120a7.png)

The three-agent architecture cost roughly 20x more. The output crossed from unusable to usable. That's the core trade Harness Engineering makes: structural overhead in exchange for reliability.

## The Retrieval Problem Inside Every Agent Harness

Both patterns — the structured `docs/` system and the Generator/Evaluator sprint cycle — share a silent dependency: the agent must find the right information from a live, evolving knowledge base when it needs it.

This is harder than it looks. Take a concrete example: the Generator is executing Sprint 3, implementing user authentication. Before writing code, it needs two kinds of information.

First, a [semantic search](https://zilliz.com/glossary/semantic-search) query: *what are this product's design principles around user sessions?* The relevant document might use "session management" or "access control" — not "user authentication." Without semantic understanding, retrieval misses it.

Second, an exact-match query: *which documents reference the `validateToken` function?* A function name is an arbitrary string with no semantic meaning. [Embedding-based retrieval](https://zilliz.com/glossary/vector-embeddings) can't reliably find it. Only keyword matching works.

These two queries happen simultaneously. They can't be separated into sequential steps.

Pure [vector search](https://zilliz.com/learn/vector-similarity-search) fails on exact match. Traditional [BM25](https://milvus.io/docs/embed-with-bm25.md) fails on semantic queries and can't predict which vocabulary a document will use. Before Milvus 2.5, the only option was two parallel retrieval systems — a vector index and a [full-text index](https://milvus.io/docs/full-text-search.md) — running concurrently at query time with custom result-fusion logic. For a live `docs/` repository with continuous updates, both indexes had to stay in sync: every document change triggered reindexing in two places, with the constant risk of inconsistency.

## How Milvus 2.6 Solves Agent Retrieval with a Single Hybrid Pipeline

Milvus is an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) designed for AI workloads. Milvus 2.6's Sparse-BM25 collapses the dual-pipeline retrieval problem into a single system.

At ingest, Milvus generates two representations simultaneously: a [dense embedding](https://zilliz.com/learn/sparse-and-dense-embeddings) for semantic retrieval and a [TF-encoded sparse vector](https://milvus.io/docs/sparse_vector.md) for BM25 scoring. Global [IDF statistics](https://zilliz.com/learn/tf-idf-understanding-term-frequency-inverse-document-frequency-in-nlp) update automatically as documents are added or removed — no manual reindex triggers. At query time, a natural-language input generates both query vector types internally. [Reciprocal Rank Fusion (RRF)](https://milvus.io/docs/rrf-ranker.md) merges the ranked results, and the caller receives a single unified result set.

![Before and after: two separate systems with manual sync, fragmented results, and custom fusion logic versus Milvus 2.6 single pipeline with dense embedding, Sparse BM25, RRF fusion, and automatic IDF maintenance producing unified results](https://assets.zilliz.com/harness_engineering_ai_agents_2_8504a6ee08.png)

One interface. One index to maintain.

On the [BEIR benchmark](https://zilliz.com/glossary/beir) — a standard evaluation suite covering 18 heterogeneous retrieval datasets — Milvus achieves 3–4x higher throughput than Elasticsearch at equivalent recall, with up to 7x QPS improvement on specific workloads. For the sprint scenario, a single query finds both the session design principle (semantic path) and every document mentioning `validateToken` (exact path). The `docs/` repository updates continuously; BM25 IDF maintenance means a newly written document participates in the next query's scoring without any batch rebuild.

This is the retrieval layer built for exactly this class of problem. When an agent harness needs to search a living knowledge base — code documentation, design decisions, sprint history — single-pipeline hybrid search isn't a nice-to-have. It's what makes the rest of the harness work.

## The Best Harness Components Are Designed to Be Deleted

Every component in a harness encodes an assumption about model limitations. Sprint decomposition was necessary when models lost coherence on long tasks. Context reset was necessary when models experienced anxiety near the window limit. Evaluator agents became necessary when self-evaluation bias was unmanageable.

These assumptions expire. Cognition's context-window trick may become unnecessary as models develop genuine long-context stamina. As models continue to improve, other components will become unnecessary overhead that slows agents down without adding reliability.

Harness Engineering isn't a fixed architecture. It's a system recalibrated with every new model release. The first question after any major upgrade isn't "what can I add?" It's "what can I remove?"

The same logic applies to retrieval. As models handle longer contexts more reliably, chunking strategies and retrieval timing will shift. Information that needs careful fragmentation today may be ingestible as full pages tomorrow. The retrieval infrastructure adapts alongside the model.

Every component in a well-built harness is waiting to be made redundant by a smarter model. That's not a problem. That's the goal.

## Get Started with Milvus

If you're building agent infrastructure that needs hybrid retrieval — semantic and keyword search in one pipeline — here's where to start:

- Read the [**Milvus 2.6 release notes**](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md) for full details on Sparse-BM25, automatic IDF maintenance, and performance benchmarks.
- Join the [**Milvus community**](https://milvus.io/community) to ask questions and share what you're building.
- [**Book a free Milvus Office Hours session**](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) to walk through your use case with a vector database expert.
- If you'd rather skip infrastructure setup, [**Zilliz Cloud**](https://cloud.zilliz.com/signup) (fully managed Milvus) offers a free tier to get started with $100 free credits upon registration with work email. 
- Star us on GitHub: [**milvus-io/milvus**](https://github.com/milvus-io/milvus) — 43k+ stars and growing.

## Frequently Asked Questions

### What is harness engineering and how is it different from prompt engineering?

Prompt engineering optimizes what you say to a model in a single exchange — phrasing, structure, examples. Harness Engineering builds the execution environment around an autonomous AI agent: the tools it can call, the knowledge it can access, the validation logic that checks its work, and the constraints that prevent architectural drift. Prompt engineering shapes one conversation turn. Harness Engineering shapes whether an agent can operate reliably for hours across hundreds of decisions without human supervision.

### Why do AI agents need both vector search and BM25 at the same time?

Agents must answer two fundamentally different retrieval queries simultaneously. Semantic queries — *what are our design principles around user sessions?* — require dense vector embeddings to match conceptually related content regardless of vocabulary. Exact-match queries — *which documents reference the `validateToken` function?* — require BM25 keyword scoring, because function names are arbitrary strings with no semantic meaning. A retrieval system that handles only one mode will systematically miss queries of the other type.

### How does Milvus Sparse-BM25 work for agent knowledge retrieval?

At ingest, Milvus generates a dense embedding and a TF-encoded sparse vector for each document simultaneously. Global IDF statistics update in real time as the knowledge base changes — no manual reindexing required. At query time, both vector types are generated internally, Reciprocal Rank Fusion merges the ranked results, and the agent receives a single unified result set. The entire pipeline runs through one interface and one index — critical for continuously updated knowledge bases like a code documentation repository.

### When should I add an evaluator agent to my agent harness?

Add a separate Evaluator when your Generator's output quality cannot be verified by automated tests alone, or when self-evaluation bias has caused missed defects. The key principle: the Evaluator must be architecturally separate from the Generator — shared context reintroduces the same bias you're trying to eliminate. The Evaluator should have access to runtime tools (browser automation, API calls, database queries) to test behavior, not just review code. Anthropic's [research](https://www.anthropic.com/engineering/harness-design-long-running-apps) found that this GAN-inspired separation moved output quality from "technically launches but broken" to "fully functional with features the solo agent never attempted."

What Is Harness Engineering for AI Agents? | Milvus


Harness Engineering: The Execution Layer AI Agents Actually Need


Claude Code Memory System Explained: 4 Layers, 5 Limits, and a Fix


We Read Claude Code's Leaked Source. Here's How Its Memory Actually Works


Vector Quantization: Beyond the TurboQuant-RaBitQ Debate


Beyond the TurboQuant-RaBitQ Debate: Why Vector Quantization Matters for AI Infrastructure Costs




When Perplexity's CTO Denis Yarats said at ASK 2026 that the company was deprioritizing MCP internally, it set off the usual cycle. YC CEO Garry Tan piled on — MCP eats too much context window, auth is broken, he built a CLI replacement in 30 minutes. Hacker News ran strongly anti-MCP.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_1_4e49d13991.png)

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_2_7dc46108c1.png)

A year ago, this level of public skepticism would have been unusual. Model Context Protocol (MCP) was positioned as the definitive standard for [AI agent](https://zilliz.com/glossary/ai-agents) tool integration. Server counts were doubling weekly. The pattern since then has followed a familiar arc: rapid hype, broad adoption, then production disillusionment.

The industry is responding fast. Bytedance's Lark/Feishu open-sourced their official CLI — 200+ commands across 11 business domains with 19 built-in Agent Skills. Google shipped gws for Google Workspace. The CLI + Skills pattern is quickly becoming the default for enterprise agent tooling, not a niche alternative.

At Zilliz, we've released [Zilliz CLI](https://docs.zilliz.com/reference/cli/overview), which lets you operate and manage [Milvus](https://milvus.io/intro) and [Zilliz Cloud](https://zilliz.com/cloud) (fully managed Milvus) directly from your terminal without leaving your coding environment. On top of that, we built [Milvus Skills](https://milvus.io/docs/milvus_for_agents.md) and [Zilliz Skill](https://docs.zilliz.com/docs/agents/zilliz-skill)s so that AI coding agents like Claude Code and Codex can manage your [vector database](https://zilliz.com/learn/what-is-vector-database) through natural language.

We also built an MCP server for Milvus and Zilliz Cloud one year ago. That experience taught us exactly where MCP breaks down — and where it still fits. Three architectural limitations pushed us toward CLI and Skills: context window bloat, passive tool design, and the inability to reuse the agent's own LLM.

In this post, we'll walk through each problem, show what we're building instead, and lay out a practical framework for choosing between MCP, CLI, and Agent Skills.

## MCP Eats 72% of Your Context Window at Startup

A standard MCP setup can consume around 72% of your available context window before the agent takes a single action. Connect three servers — GitHub, Playwright, and an IDE integration — on a 200K-token model, and tool definitions alone occupy roughly 143K tokens. The agent hasn't done anything yet. It's already three-quarters full.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_3_767d46c583.png)

The cost isn't just tokens. The more unrelated content packed into context, the weaker the model's focus on what actually matters. A hundred tool schemas sitting in context means the agent wades through all of them on every decision. Researchers have documented what they call _context rot_ — degraded reasoning quality from context overload. In measured tests, tool selection accuracy dropped from 43% to below 14% as tool count increased. More tools, paradoxically, means worse tool use.

The root cause is architectural. MCP loads all tool descriptions in full at session start, regardless of whether the current conversation will ever use them. That's a protocol-level design choice, not a bug — but the cost scales with every tool you add.

Agent skills take a different approach: **progressive disclosure**. At session start, an agent reads only each Skill's metadata — name, one-line description, trigger condition. A few dozen tokens total. The full Skill content loads only when the agent determines it's relevant. Think of it this way: MCP lines up every tool at the door and makes you choose; Skills gives you an index first, full content on demand.

CLI tools offer a similar advantage. An agent runs git --help or docker --help to discover capabilities on demand, without preloading every parameter definition. Context cost is pay-as-you-go, not upfront.

At a small scale, the difference is negligible. At production scale, it's the difference between an agent that works and one that drowns in its own tool definitions.

## MCP's Passive Architecture Limits Agent Workflows

MCP is a tool-calling protocol: how to discover tools, call them, and receive results. Clean design for simple use cases. But that cleanness is also a constraint.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_4_f80de07814.png)

### Flat Tool Space with No Hierarchy

An MCP tool is a flat function signature. No subcommands, no awareness of session lifecycle, no sense of where the agent is in a multi-step workflow. It waits to be called. That's all it does.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_5_e7f3630e1f.png)

A CLI works differently. git commit, git push, and git log are completely different execution paths sharing a single interface. An agent runs --help, explores the available surface incrementally, and expands only what it needs — without front-loading all the parameter documentation into context.

### Skills Encode Workflow Logic — MCP Can't

An Agent Skill is a Markdown file containing a standard operating procedure: what to do first, what to do next, how to handle failures, and when to surface something to the user. The agent receives not just a tool but an entire workflow. Skills actively shape how an agent behaves during a conversation — what triggers them, what they prepare in advance, and how they recover from errors. MCP tools can only wait.

### MCP Can't Access the Agent's LLM

This is the limitation that actually stopped us.

When we built [claude-context](https://github.com/zilliztech/claude-context) — an MCP plugin that adds [semantic search](https://zilliz.com/glossary/semantic-search) to Claude Code and other AI coding agents, giving them deep context from an entire codebase — we wanted to retrieve relevant historical conversation snippets from Milvus and surface them as context. The [vector search](https://zilliz.com/learn/vector-similarity-search) retrieval worked. The problem was what to do with the results.

Retrieve the top 10 results, and maybe 3 are useful. The other 7 are noise. Hand all 10 to the outer agent, and the noise interferes with the answer. In testing, we saw responses get distracted by irrelevant historical records. We needed to filter before passing results up.

We tried several approaches. Adding a reranking step inside the MCP server using a small model: not accurate enough, and the relevance threshold needed per-use-case tuning. Using a large model for reranking: technically sound, but an MCP server runs as a separate process with no access to the outer agent's LLM. We'd have to configure a separate LLM client, manage a separate API key, and handle a separate call path.

What we wanted was simple: let the outer agent's LLM participate directly in the filtering decision. Retrieve the top 10, let the agent itself judge what's worth keeping, and return only the relevant results. No second model. No extra API keys.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_6_aca200f359.png)

MCP can't do this. The process boundary between server and agent is also an intelligence boundary. The server can't use the agent's LLM; the agent can't govern what happens inside the server. Fine for simple CRUD tools. The moment a tool needs to make a judgment call, that isolation becomes a real constraint.

An Agent Skill solves this directly. A retrieval Skill can call vector search for the top 10, have the agent's own LLM assess relevance, and return only what passes. No additional model. The agent does the filtering itself.

## What We Built Instead with CLI and Skills

We see CLI + Skills as the direction for agent-tool interaction — not just for memory retrieval, but across the stack. This conviction drives everything we're building.

### memsearch: A Skills-Based Memory Layer for AI Agents

We built [memsearch](https://github.com/zilliztech/memsearch), an open-source memory layer for Claude Code and other AI agents. The Skill runs inside a subagent with three stages: Milvus handles the initial vector search for broad discovery, the agent's own LLM evaluates relevance and expands context for promising hits, and a final drill-down accesses original conversations only when needed. Noise gets discarded at each stage — intermediate retrieval junk never reaches the primary context window.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_7_7c85103513.png)

The key insight: the agent's intelligence is part of the tool's execution. The LLM already in the loop does the filtering — no second model, no extra API key, no brittle threshold tuning. This is a specific use case — conversation-context retrieval for coding agents — but the architecture generalizes to any scenario where a tool needs judgment, not just execution.

### Zilliz CLI, Skills, and Plugin for Vector Database Operations

Milvus is the world's most widely adopted open-source vector database with [43K+ stars on GitHub](https://github.com/milvus-io/milvus). [Zilliz Cloud](https://zilliz.com/cloud) is the fully managed service of Milvus with advanced enterprise features and is much faster than Milvus.

The same layered architecture mentioned above drives our developer tools:

-   [Zilliz CLI](https://docs.zilliz.com/reference/cli/overview) is the infrastructure layer. Cluster management, [collection operations](https://milvus.io/docs/manage-collections.md), vector search, [RBAC](https://milvus.io/docs/rbac.md), backups, billing — everything you'd do in the Zilliz Cloud console, available from the terminal. Humans and agents use the same commands. Zilliz CLI also serves as the foundation for Milvus Skills and Zilliz Skills.
-   [Milvus Skill](https://milvus.io/docs/milvus_for_agents.md) is the knowledge layer for open-source Milvus. It teaches AI coding agents (Claude Code, Cursor, Codex, GitHub Copilot) to operate any Milvus deployment — [Milvus Lite](https://milvus.io/docs/milvus_lite.md), Standalone, or Distributed — through [pymilvus](https://milvus.io/docs/install-pymilvus.md) Python code: connections, [schema design](https://milvus.io/docs/schema-hands-on.md), CRUD, [hybrid search](https://zilliz.com/learn/hybrid-search-with-milvus), [full-text search](https://milvus.io/docs/full-text-search.md), [RAG pipelines](https://zilliz.com/learn/Retrieval-Augmented-Generation).
-   [Zilliz Skill](https://docs.zilliz.com/docs/agents/zilliz-skill) does the same for Zilliz Cloud, teaching agents to manage cloud infrastructure through Zilliz CLI.
-   [Zilliz Plugin](https://github.com/zilliztech/zilliz-plugin) is the developer experience layer for Claude Code — wraps CLI + Skill into a guided experience with slash commands like /zilliz:quickstart and /zilliz:status.

CLI handles execution, Skills encode knowledge and workflow logic, Plugin delivers the UX. No MCP server in the loop.

For more details, check out these resources:

-   [Introducing Zilliz CLI and Agent Skills for Zilliz Cloud](https://zilliz.com/blog/introducing-zilliz-cli-and-agent-skills-for-zilliz-cloud)
-   [Zilliz Cloud Just Landed in Claude Code](https://zilliz.com/blog/zilliz-cloud-just-landed-in-claude-code)
-   [AI Prompts — Zilliz Cloud Developer Hub](https://docs.zilliz.com/docs/agents/zilliz-ai-prompts)
-   [Zilliz CLI Reference — Zilliz Cloud Developer Hub](https://docs.zilliz.com/reference/cli/overview)
-   [Zilliz Skill — Zilliz Cloud Developer Hub](https://docs.zilliz.com/docs/agents/zilliz-skill)
-   [Milvus for AI Agents — Milvus Documentation](https://milvus.io/docs/milvus_for_agents.md)

## Is MCP Actually Dying?

A lot of developers and companies including us here at Zilliz are turning to CLI and Skills. But is MCP really dying?

The short answer: no — but its scope is narrowing to where it actually fits.

MCP has been donated to the Linux Foundation. Active servers number over 10,000. SDK monthly downloads sit at 97 million. An ecosystem that size doesn't disappear because of a conference comment.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_8_b2246e6825.png)

A Hacker News thread — _"When does MCP make sense vs CLI?"_ — drew responses that mostly favored CLI: "CLI tools are like precision instruments," "CLIs also feel snappier than MCPs." Some developers hold a more balanced view: Skills are a detailed recipe that helps you solve a problem better; MCP is the tool that helps you solve the problem. Both have their place.

That's fair — but it raises a practical question. If the recipe itself can direct the agent on which tools to use and how, is a separate tool-distribution protocol still necessary?

It depends on the use case.

![](https://assets.zilliz.com/is_mcp_dead_cli_and_skills_for_ai_agents_md_9_e2cb28812b.png)

**MCP over stdio** — the version most developers run locally — is where the problems accumulate: unstable inter-process communication, messy environment isolation, high token overhead. In that context, better alternatives exist for almost every use case.

**MCP over HTTP** is a different story. Enterprise internal tooling platforms need centralized permission management, unified OAuth, standardized telemetry and logging. Fragmented CLI tools genuinely struggle to provide these. MCP's centralized architecture has real value in that context.

What Perplexity actually dropped was primarily the stdio use case. Denis Yarats specified "internally" and didn't call for industry-wide adoption of that choice. That nuance got lost in transmission — "Perplexity abandons MCP" spreads considerably faster than "Perplexity deprioritizes MCP over stdio for internal tool integration."

MCP emerged because it solved a real problem: before it, every AI application wrote its own tool-calling logic, with no shared standard. MCP provided a unified interface at the right moment, and the ecosystem built quickly. Production experience then surfaced the limitations. That's a normal arc for infrastructure tooling — not a death sentence.

## When to Use MCP, CLI, or Skills

|   | MCP over stdio (Local) | MCP over HTTP (Enterprise) |
| --- | --- | --- |
| **Authentication** | None | OAuth, centralized |
| **Connection stability** | Process isolation issues | Stable HTTPS |
| **Logging** | No standard mechanism | Centralized telemetry |
| **Access control** | None | Role-based permissions |
| **Our take** | Replace with CLI + Skills | Keep for enterprise tooling |

For teams choosing their [agentic AI](https://zilliz.com/glossary/ai-agents) tooling stack, here's how the layers fit:

| Layer | What It Does | Best For | Examples |
| --- | --- | --- | --- |
| **CLI** | Operational tasks, infra management | Commands that agents and humans both run | git, docker, zilliz-cli |
| **Skills** | Agent workflow logic, encoded knowledge | Tasks needing LLM judgment, multi-step SOPs | milvus-skill, zilliz-skill, memsearch |
| **REST APIs** | External integrations | Connecting to third-party services | GitHub API, Slack API |
| **MCP HTTP** | Enterprise tool platforms | Centralized auth, audit logging | Internal tool gateways |

## Get Started

Everything we've discussed in this article is available today:

-   [**memsearch**](https://github.com/zilliztech/memsearch) — the Skills-based memory layer for AI agents. Drop it into Claude Code or any agent that supports Skills.
-   [**Zilliz CLI**](https://docs.zilliz.com/reference/cli/overview) — manage Milvus and Zilliz Cloud from your terminal. Install it and explore the subcommands your agents can use.
-   [**Milvus Skill**](https://milvus.io/docs/milvus_for_agents.md) and [**Zilliz Skill**](https://docs.zilliz.com/docs/agents/zilliz-skill) — give your AI coding agent native Milvus and Zilliz Cloud knowledge.

Have questions about vector search, agent architecture, or building with CLI and Skills? Join the [Milvus Discord community](https://discord.com/invite/8uyFbECzPX) or [book a free Office Hours session](https://milvus.io/office-hours) to talk through your use case.

Ready to build? [Sign up for Zilliz Cloud](https://cloud.zilliz.com/signup) — new accounts with a work email get $100 in free credits. Already have an account? [Sign in here](https://cloud.zilliz.com/login).

## Frequently Asked Questions

### What is wrong with MCP for AI agents?

MCP has three main architectural limitations in production. First, it loads all tool schemas into the context window at session start — connecting just three MCP servers on a 200K-token model can consume over 70% of available context before the agent does anything. Second, MCP tools are passive: they wait to be called and can't encode multi-step workflows, error-handling logic, or standard operating procedures. Third, MCP servers run as separate processes with no access to the agent's LLM, so any tool that needs judgment (like filtering search results for relevance) requires configuring a separate model with its own API key. These problems are most acute with MCP over stdio; MCP over HTTP mitigates some of them.

### What is the difference between MCP and Agent Skills?

MCP is a tool-calling protocol that defines how an agent discovers and invokes external tools. An Agent Skill is a Markdown file containing a full standard operating procedure — triggers, step-by-step instructions, error handling, and escalation rules. The key architectural difference: Skills run inside the agent's process, so they can leverage the agent's own LLM for judgment calls like relevance filtering or result reranking. MCP tools run in a separate process and can't access the agent's intelligence. Skills also use progressive disclosure — only lightweight metadata loads at startup, with full content loading on demand — keeping context window usage minimal compared to MCP's upfront schema loading.

### When should I still use MCP instead of CLI or Skills?

MCP over HTTP still makes sense for enterprise tooling platforms where you need centralized OAuth, role-based access control, standardized telemetry, and audit logging across many internal tools. Fragmented CLI tools struggle to provide these enterprise requirements consistently. For local development workflows — where agents interact with tools on your machine — CLI + Skills typically offers better performance, lower context overhead, and more flexible workflow logic than MCP over stdio.

### How do CLI tools and Agent Skills work together?

CLI provides the execution layer (the actual commands), while Skills provide the knowledge layer (when to run which commands, in what order, and how to handle failures). For example, Zilliz CLI handles infrastructure operations like cluster management, collection CRUD, and vector search. Milvus Skill teaches the agent the right pymilvus patterns for schema design, hybrid search, and RAG pipelines. The CLI does the work; the Skill knows the workflow. This layered pattern — CLI for execution, Skills for knowledge, a plugin for UX — is how we've structured all of our developer tooling at Zilliz.

### MCP vs Skills vs CLI: when should I use each?

CLI tools like git, docker, or zilliz-cli are best for operational tasks — they expose hierarchical subcommands and load on demand. Skills like milvus-skill are best for agent workflow logic — they carry operating procedures, error recovery, and can access the agent's LLM. MCP over HTTP still fits enterprise tool platforms needing centralized OAuth, permissions, and audit logging. MCP over stdio — the local version — is being replaced by CLI + Skills in most production setups.

Is MCP Dead? MCP vs CLI vs Agent Skills Compared


Is MCP Dead? What We Learned Building with MCP, CLI, and Agent Skills


Best Embedding Model for RAG 2026: 10 Models Compared


How to Choose the Best Embedding Model for RAG in 2026: 10 Models Benchmarked



Here's a story that's more common than it should be: a QA engineer runs a cleanup script against what they think is the staging environment. Except the connection string points to production. A few seconds later, core vector collections are gone — feature data lost, [similarity search](https://zilliz.com/glossary/similarity-search) returning empty results, services degrading across the board. The postmortem finds the same root cause it always does: everyone was connecting as `root`, there were no access boundaries, and nothing stopped a test account from dropping production data.

This isn't a one-off. Teams building on [Milvus](https://milvus.io/) — and [vector databases](https://zilliz.com/learn/what-is-a-vector-database) in general — tend to focus on [index performance](https://zilliz.com/learn/vector-index), throughput, and data scale, while treating access control as something to deal with later. But "later" usually arrives in the form of an incident. As Milvus moves from prototype to the backbone of production [RAG pipelines](https://zilliz.com/learn/Retrieval-Augmented-Generation), recommendation engines, and real-time [vector search](https://zilliz.com/learn/what-is-vector-search), the question becomes unavoidable: who can access your Milvus cluster, and what exactly are they allowed to do?

Milvus includes a built-in RBAC system to answer that question. This guide covers what RBAC is, how Milvus implements it, and how to design an access control model that keeps production safe — complete with code examples and a full RAG system walkthrough.

## What Is RBAC (Role-Based Access Control)?

**Role-Based Access Control (RBAC)** is a security model where permissions are not assigned directly to individual users. Instead, permissions are grouped into roles, and users are assigned one or more roles. A user's effective access is the union of all permissions from their assigned roles. RBAC is the standard access control model in production database systems — PostgreSQL, MySQL, MongoDB, and most cloud services use it.

RBAC solves a fundamental scaling problem: when you have dozens of users and services, managing permissions per-user becomes unmaintainable. With RBAC, you define a role once (e.g., "read-only on collection X"), assign it to ten services, and update it in one place when requirements change.

## How Does Milvus Implement RBAC?

Milvus RBAC is built on four concepts:

| Concept | What It Is | Example |
|---|---|---|
| **Resource** | The thing being accessed | A [Milvus instance](https://milvus.io/docs/architecture_overview.md), a [database](https://milvus.io/docs/manage-databases.md), or a specific collection |
| **Privilege / Privilege Group** | The action being performed | `Search`, `Insert`, `DropCollection`, or a group like `COLL_RO` (collection read-only) |
| **Role** | A named set of privileges scoped to resources | `role_read_only`: can search and query all collections in `default` database |
| **User** | A Milvus account (human or service) | `rag_writer`: the service account used by the ingestion pipeline |

Access is never assigned directly to users. Users get roles, roles contain privileges, and privileges are scoped to resources. This is the same [RBAC model](https://zilliz.com/blog/milvus-2-5-rbac-enhancements) used in most production database systems. If ten users share the same role, you update the role once and the change applies to all of them.

![Milvus RBAC model showing how Users are assigned to Roles, and Roles contain Privileges and Privilege Groups that apply to Resources](https://assets.zilliz.com/milvus_access_control_rbac_guide_1_c872ea8932.png)

When a request hits Milvus, it goes through three checks:

1. **Authentication** — is this a valid user with correct credentials?
2. **Role check** — does this user have at least one role assigned?
3. **Privilege check** — does any of the user's roles grant the requested action on the requested resource?

If any check fails, the request is rejected.

![Milvus authentication and authorization flow: Client Request goes through Authentication, Role Check, and Privilege Check — rejected at any failed step, executed only if all pass](https://assets.zilliz.com/milvus_access_control_rbac_guide_2_2275b37ea6.png)

## How to Enable Authentication in Milvus

By default, Milvus runs with authentication disabled — every connection has full access. The first step is turning it on.

### Docker Compose

Edit `milvus.yaml` and set `authorizationEnabled` to `true`:

```
common:
  security:
    authorizationEnabled: true
```

### Helm Charts

Edit `values.yaml` and add the setting under `extraConfigFiles`:

```
extraConfigFiles:
  user.yaml: |+
    common:
      security:
        authorizationEnabled: true
```

For [Milvus Operator](https://milvus.io/docs/install_cluster-milvusoperator.md) deployments on [Kubernetes](https://milvus.io/docs/prerequisite-helm.md), the same configuration goes into the Milvus CR's `spec.config` section.

Once authentication is enabled and Milvus restarts, every connection must provide credentials. Milvus creates a default `root` user with the password `Milvus` — change this immediately.

```
from pymilvus import MilvusClient

# Connect with the default root account
client = MilvusClient(
    uri='http://localhost:19530',
    token="root:Milvus"
)

# Change the password
client.update_password(
    user_name="root",
    old_password="Milvus",
    new_password="xgOoLudt3Kc#Pq68"
)
```

## How to Configure Users, Roles, and Privileges

With authentication enabled, here's the typical setup workflow.

### Step 1: Create Users

Don't let services or team members use `root`. Create dedicated accounts for each user or service.

```
client.create_user(user_name="user_1", password="P@ssw0rd")
```

### Step 2: Create Roles

Milvus has a built-in `admin` role, but in practice you'll want custom roles that match your actual access patterns.

```
client.create_role(role_name="role_a")
```

### Step 3: Create Privilege Groups

A privilege group bundles multiple privileges under one name, making it easier to manage access at scale. Milvus provides 9 built-in privilege groups:

| Built-in Group | Scope | What It Allows |
|---|---|---|
| `COLL_RO` | Collection | Read-only operations (Query, Search, etc.) |
| `COLL_RW` | Collection | Read and write operations |
| `COLL_Admin` | Collection | Full collection management |
| `DB_RO` | Database | Read-only database operations |
| `DB_RW` | Database | Read and write database operations |
| `DB_Admin` | Database | Full database management |
| `Cluster_RO` | Cluster | Read-only cluster operations |
| `Cluster_RW` | Cluster | Read and write cluster operations |
| `Cluster_Admin` | Cluster | Full cluster management |

You can also create custom privilege groups when the built-in ones don't fit:

```
# Create a privilege group
client.create_privilege_group(group_name='privilege_group_1')

# Add privileges to the group
client.add_privileges_to_group(
    group_name='privilege_group_1',
    privileges=['Query', 'Search']
)
```

### Step 4: Grant Privileges to a Role

Grant individual privileges or privilege groups to a role, scoped to specific resources. The `collection_name` and `db_name` parameters control the scope — use `*` for all.

```
# Grant a single privilege
client.grant_privilege_v2(
    role_name="role_a",
    privilege="Search",
    collection_name='collection_01',
    db_name='default',
)

# Grant a privilege group
client.grant_privilege_v2(
    role_name="role_a",
    privilege="privilege_group_1",
    collection_name='collection_01',
    db_name='default',
)

# Grant a cluster-level privilege (* means all resources)
client.grant_privilege_v2(
    role_name="role_a",
    privilege="ClusterReadOnly",
    collection_name='*',
    db_name='*',
)
```

### Step 5: Assign Roles to Users

A user can hold multiple roles. Their effective permissions are the union of all assigned roles.

```
client.grant_role(user_name="user_1", role_name="role_a")
```

## How to Audit and Revoke Access

Knowing what access exists is just as important as granting it. Stale permissions — from former team members, retired services, or one-off debugging sessions — accumulate silently and widen the attack surface.

### Check Current Permissions

View a user's assigned roles:

```
client.describe_user(user_name="user_1")
```

View a role's granted privileges:

```
client.describe_role(role_name="role_a")
```

### Revoke Privileges from a Role

```
# Remove a single privilege
client.revoke_privilege_v2(
    role_name="role_a",
    privilege="Search",
    collection_name='collection_01',
    db_name='default',
)

# Remove a privilege group
client.revoke_privilege_v2(
    role_name="role_a",
    privilege="privilege_group_1",
    collection_name='collection_01',
    db_name='default',
)
```

### Unassign a Role from a User

```
client.revoke_role(
    user_name='user_1',
    role_name='role_a'
)
```

### Delete Users or Roles

Remove all role assignments before deleting a user, and revoke all privileges before dropping a role:

```
client.drop_user(user_name="user_1")
client.drop_role(role_name="role_a")
```

## Example: How to Design RBAC for a Production RAG System

Abstract concepts click faster with a concrete example. Consider a [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) system built on Milvus with three distinct services:

| Service | Responsibility | Required Access |
|---|---|---|
| **Platform admin** | Manages the Milvus cluster — creates collections, monitors health, handles upgrades | Full cluster admin |
| **Ingestion service** | Generates [vector embeddings](https://zilliz.com/glossary/vector-embeddings) from documents and writes them to collections | Read + write on collections |
| **Search service** | Handles [vector search](https://zilliz.com/learn/what-is-vector-search) queries from end users | Read-only on collections |

Here's a complete setup using [PyMilvus](https://milvus.io/docs/install-pymilvus.md):

```
from pymilvus import MilvusClient

client = MilvusClient(
    uri='http://localhost:19530',
    token="root:xxx"  # Replace with your updated root password
)

# 1. Create users
client.create_user(user_name="rag_admin", password="xxx")
client.create_user(user_name="rag_reader", password="xxx")
client.create_user(user_name="rag_writer", password="xxx")

# 2. Create roles
client.create_role(role_name="role_admin")
client.create_role(role_name="role_read_only")
client.create_role(role_name="role_read_write")

# 3. Grant access to roles

# Admin role: cluster-level admin access
client.grant_privilege_v2(
    role_name="role_admin",
    privilege="Cluster_Admin",
    collection_name='*',
    db_name='*',
)

# Read-only role: collection-level read-only access
client.grant_privilege_v2(
    role_name="role_read_only",
    privilege="COLL_RO",
    collection_name='*',
    db_name='default',
)

# Read-write role: collection-level read and write access
client.grant_privilege_v2(
    role_name="role_read_write",
    privilege="COLL_RW",
    collection_name='*',
    db_name='default',
)

# 4. Assign roles to users
client.grant_role(user_name="rag_admin", role_name="role_admin")
client.grant_role(user_name="rag_reader", role_name="role_read_only")
client.grant_role(user_name="rag_writer", role_name="role_read_write")
```

Each service gets exactly the access it needs. The search service can't accidentally delete data. The ingestion service can't modify cluster settings. And if the search service's credentials leak, the attacker can read [embedding vectors](https://zilliz.com/glossary/vector-embeddings) — but can't write, delete, or escalate to admin.

For teams managing access across multiple Milvus deployments, [Zilliz Cloud](https://cloud.zilliz.com/signup) (managed Milvus) provides built-in RBAC with a web console for managing users, roles, and permissions — no scripting required. Useful when you'd rather manage access through a UI than maintain setup scripts across environments.

## Access Control Best Practices for Production

The setup steps above are the mechanics. Here are the design principles that keep access control effective over time.

### Lock Down the Root Account

Change the default `root` password before anything else. In production, the root account should be used only for emergency operations and stored in a secrets manager — not hardcoded in application configs or shared over Slack.

### Separate Environments Completely

Use different [Milvus instances](https://milvus.io/docs/architecture_overview.md) for development, staging, and production. Environment separation by RBAC alone is fragile — one misconfigured connection string and a dev service is writing to production data. Physical separation (different clusters, different credentials) eliminates this class of incident entirely.

### Apply Least Privilege

Give each user and service the minimum access needed to do its job. Start narrow and widen only when there's a specific, documented need. In development environments you can be more relaxed, but production access should be strict and reviewed regularly.

### Clean Up Stale Access

When someone leaves the team or a service gets decommissioned, revoke their roles and delete their accounts immediately. Unused accounts with active permissions are the most common vector for unauthorized access — they're valid credentials that nobody is monitoring.

### Scope Privileges to Specific Collections

Avoid granting `collection_name='*'` unless the role genuinely needs access to every collection. In multi-tenant setups or systems with multiple data pipelines, scope each role to only the [collections](https://milvus.io/docs/manage-collections.md) it operates on. This limits the blast radius if credentials are compromised.

---

If you're deploying [Milvus](https://milvus.io/) in production and working through access control, security, or multi-tenant design, we'd love to help:

- Join the [Milvus Slack community](https://slack.milvus.io/) to discuss real deployment practices with other engineers running Milvus at scale.
- [Book a free 20-minute Milvus Office Hours session](https://milvus.io/office-hours) to walk through your RBAC design — whether it's role structure, collection-level scoping, or multi-environment security.
- If you'd rather skip the infrastructure setup and manage access control through a UI, [Zilliz Cloud](https://cloud.zilliz.com/signup) (managed Milvus) includes built-in RBAC with a web console — plus [encryption](https://zilliz.com/cloud-security), network isolation, and SOC 2 compliance out of the box.

---

A few questions that come up when teams start configuring access control in Milvus:

**Q: Can I restrict a user to only specific collections, not all of them?**

Yes. When you call [`grant_privilege_v2`](https://milvus.io/docs/grant_privilege.md), set `collection_name` to the specific collection rather than `*`. The user's role will only have access to that collection. You can grant the same role privileges on multiple collections by calling the function once per collection.

**Q: What's the difference between a privilege and a privilege group in Milvus?**

A privilege is a single action like `Search`, `Insert`, or `DropCollection`. A [privilege group](https://milvus.io/docs/privilege_group.md) bundles multiple privileges under one name — for example, `COLL_RO` includes all read-only collection operations. Granting a privilege group is functionally the same as granting each of its constituent privileges individually, but easier to manage.

**Q: Does enabling authentication affect Milvus query performance?**

The overhead is negligible. Milvus validates credentials and checks role permissions on each request, but this is an in-memory lookup — it adds microseconds, not milliseconds. There is no measurable impact on [search](https://milvus.io/docs/single-vector-search.md) or [insert](https://milvus.io/docs/insert-update-delete.md) latency.

**Q: Can I use Milvus RBAC in a multi-tenant setup?**

Yes. Create separate roles per tenant, scope each role's privileges to that tenant's collections, and assign the corresponding role to each tenant's service account. This gives you collection-level isolation without needing separate Milvus instances. For larger-scale multi-tenancy, see the [Milvus multi-tenancy guide](https://milvus.io/docs/multi_tenancy.md).


Milvus Access Control: Configure RBAC for Production


Milvus Access Control Guide: How to Configure RBAC for Production


Vector Database High Availability: Milvus CDC Standby Cluster Guide


Vector Database High Availability: How to Build a Milvus Standby Cluster with CDC



Your [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) pipeline retrieves documents for every query, regardless of whether retrieval is needed. It runs the same similarity search on code, natural language, and financial reports. And when results are bad, you have no way to tell which stage broke.

These are symptoms of naive RAG—a fixed pipeline that treats every query the same way. Modern RAG systems work differently. They route queries to the right handler, combine multiple retrieval methods, and evaluate each stage independently.

This article walks through a four-node architecture for building smarter RAG systems, explains how to implement [hybrid retrieval](https://zilliz.com/learn/hybrid-search-a-practical-guide) without maintaining separate indexes, and shows how to evaluate each pipeline stage so you can debug problems faster.

## Why Long Context Doesn't Replace RAG

"Just put everything in the prompt" is a common suggestion now that models support 128K+ token windows. It doesn't hold up in production for two reasons.

**Cost scales with your knowledge base, not your query.** Every request sends the full knowledge base through the model. For a 100K-token corpus, that's 100K input tokens per request—regardless of whether the answer requires one paragraph or ten. Monthly inference costs grow linearly with corpus size.

**Attention degrades with context length.** Models struggle to focus on relevant information buried in long contexts. Research on the "lost in the middle" effect (Liu et al., 2023) shows that models are more likely to miss information placed in the middle of long inputs. Larger context windows haven't solved this—attention quality hasn't kept pace with window size.

RAG avoids both problems by retrieving only the relevant passages before generation. The question isn't whether RAG is needed—it's how to build RAG that actually works.

## What's Wrong with Traditional RAG?

Traditional RAG follows a fixed pipeline: embed the query, run [vector similarity search](https://zilliz.com/learn/what-is-vector-search), take the top-K results, generate an answer. Every query follows the same path.

This creates two problems:

1. **Wasted compute on trivial queries.** "What is 2 + 2?" doesn't need retrieval, but the system runs it anyway—adding latency and cost for no benefit.

2. **Brittle retrieval on complex queries.** Ambiguous phrasing, synonyms, or mixed-language queries often defeat pure vector similarity. When retrieval misses relevant documents, generation quality drops with no fallback.

The fix: add decision-making before retrieval. A modern RAG system decides *whether* to retrieve, *what* to search for, and *how* to search—rather than blindly running the same pipeline every time.

## How Modern RAG Systems Work: A Four-Node Architecture

![](https://assets.zilliz.com/build_smarter_rag_routing_hybrid_retrieval_1_058ab7eb6b.png)

Instead of a fixed pipeline, a modern RAG system routes each query through four decision nodes. Each node answers one question about how to handle the current query.

### Node 1: Query Routing — Does This Query Need Retrieval?

Query routing is the first decision in the pipeline. It classifies the incoming query and sends it down the appropriate path:

| Query Type | Example | Action |
|---|---|---|
| Common-sense / general knowledge | "What is 2 + 2?" | Answer directly with the LLM—skip retrieval |
| Knowledge-base question | "What are the specs for Model X?" | Route to the retrieval pipeline |
| Real-time information | "Weather in Paris this weekend" | Call an external API |

Routing upfront avoids unnecessary retrieval for queries that don't need it. In systems where a large share of queries are simple or general-knowledge, this alone can cut compute costs significantly.

### Node 2: Query Rewriting — What Should the System Search For?

User queries are often vague. A question like "the main numbers in LightOn's Q3 report" doesn't translate well into a search query.

Query rewriting transforms the original question into structured search conditions:

- **Time range:** July 1 – September 30, 2025 (Q3)
- **Document type:** Financial report
- **Entity:** LightOn, Finance department

This step bridges the gap between how users ask questions and how retrieval systems index documents. Better queries mean fewer irrelevant results.

### Node 3: Retrieval Strategy Selection — How Should the System Search?

Different content types need different retrieval strategies. A single method can't cover everything:

| Content Type | Best Retrieval Method | Why |
|---|---|---|
| Code (variable names, function signatures) | Lexical search ([BM25](https://zilliz.com/learn/mastering-bm25-a-deep-dive-into-the-algorithm-and-application-in-milvus)) | Exact keyword matching works well on structured tokens |
| Natural language (docs, articles) | Semantic search (dense vectors) | Handles synonyms, paraphrases, and intent |
| Multimodal (charts, diagrams, drawings) | Multimodal retrieval | Captures visual structure that text extraction misses |

Documents are tagged with metadata at indexing time. At query time, these tags guide both which documents to search and which retrieval method to use.

### Node 4: Minimal-Context Generation — How Much Context Does the Model Need?

After retrieval and [reranking](https://zilliz.com/learn/optimize-rag-with-rerankers-the-role-and-tradeoffs), the system sends only the most relevant passages to the model—not entire documents.

This matters more than it sounds. Compared with full-document loading, passing only the relevant passages can reduce token usage by over 90%. Lower token counts mean faster responses and lower costs, even when caching is in play.

## Why Hybrid Retrieval Matters for Enterprise RAG

In practice, retrieval strategy selection (Node 3) is where most teams get stuck. No single retrieval method covers all enterprise document types.

Some argue that keyword search is sufficient—after all, Claude Code's grep-based code search works well. But code is highly structured, with consistent naming conventions. Enterprise documents are a different story.

### Enterprise Documents Are Messy

**Synonyms and varied phrasing.** "Optimize memory usage" and "reduce memory footprint" mean the same thing but use different words. Keyword search matches one and misses the other. In multilingual environments—Chinese with word segmentation, Japanese with mixed scripts, German with compound words—the problem multiplies.

**Visual structure matters.** Engineering drawings depend on layout. Financial reports rely on tables. Medical images depend on spatial relationships. OCR extracts text but loses structure. Text-only retrieval can't handle these documents reliably.

### How to Implement Hybrid Retrieval

Hybrid retrieval combines multiple search methods—typically [BM25 for keyword matching and dense vectors for semantic search](https://zilliz.com/learn/sparse-and-dense-embeddings)—to cover what neither method handles alone.

![](https://assets.zilliz.com/build_smarter_rag_routing_hybrid_retrieval_2_7f305f024e.png)

The traditional approach runs two separate systems: one for BM25, one for vector search. Each query hits both, and results are merged afterward. It works, but it comes with real overhead:

| | Traditional (Separate Systems) | Unified (Single Collection) |
|---|---|---|
| Storage | Two separate indexes | One collection, both vector types |
| Data sync | Must keep two systems in sync | Single write path |
| Query path | Two queries + result merging | One API call, automatic fusion |
| Tuning | Adjust merge weights across systems | Change dense/sparse weight in one query |
| Operational complexity | High | Low |

[Milvus](https://milvus.io/) 2.6 supports both dense vectors (for semantic search) and sparse vectors (for BM25-style keyword search) in the same collection. A single API call returns fused results, with retrieval behavior adjustable by changing the weight between vector types. No separate indexes, no sync issues, no merge latency.

## How to Evaluate a RAG Pipeline Stage by Stage

Checking only the final answer isn't enough. RAG is a multi-stage pipeline, and a failure at any stage propagates downstream. If you only measure answer quality, you can't tell whether the problem is in routing, rewriting, retrieval, reranking, or generation.

When users report "inaccurate results," the root cause could be anywhere: routing may skip retrieval when it shouldn't; query rewriting may drop key entities; retrieval may miss relevant documents; reranking may bury good results; or the model may ignore the retrieved context entirely.

Evaluate each stage with its own metrics:

| Stage | Metric | What It Catches |
|---|---|---|
| Routing | F1 score | High false-negative rate = queries needing retrieval are skipped |
| Query rewriting | Entity extraction accuracy, synonym coverage | Rewritten query drops important terms or changes intent |
| Retrieval | Recall@K, NDCG@10 | Relevant documents aren't retrieved, or are ranked too low |
| Reranking | Precision@3 | Top results aren't actually relevant |
| Generation | Faithfulness, answer completeness | Model ignores retrieved context or gives partial answers |

**Set up layered monitoring.** Use offline test sets to define baseline metric ranges for each stage. In production, trigger alerts when any stage drops below its baseline. This lets you catch regressions early and trace them to a specific stage—instead of guessing.

## What to Build First

Three priorities stand out from real-world RAG deployments:

1. **Add routing early.** Many queries don't need retrieval at all. Filtering them upfront reduces load and improves response time with minimal engineering effort.

2. **Use unified hybrid retrieval.** Maintaining separate BM25 and vector search systems doubles storage costs, creates sync complexity, and adds merge latency. A unified system like Milvus 2.6—where dense and sparse vectors live in the same collection—eliminates these issues.

3. **Evaluate each stage independently.** End-to-end answer quality alone is not a useful signal. Per-stage metrics (F1 for routing, Recall@K and NDCG for retrieval) let you debug faster and avoid breaking one stage while tuning another.

The real value of a modern RAG system isn't just retrieval—it's knowing *when* to retrieve and *how* to retrieve. Start with routing and unified hybrid search, and you'll have a foundation that scales.

---

If you're building or upgrading a RAG system and running into retrieval quality issues, we'd love to help:

- Join the [Milvus Slack community](https://slack.milvus.io/) to ask questions, share your architecture, and learn from other developers working on similar problems.
- [Book a free 20-minute Milvus Office Hours session](https://milvus.io/office-hours) to walk through your use case—whether it's routing design, hybrid retrieval setup, or multi-stage evaluation.
- If you'd rather skip the infrastructure setup, [Zilliz Cloud](https://cloud.zilliz.com/signup) (managed Milvus) offers a free tier to get started.

---

A few questions that come up often when teams start building smarter RAG systems:

**Q: Is RAG still necessary now that models support 128K+ context windows?**

Yes. Long context windows help when you need to process a single large document, but they don't replace retrieval for knowledge-base queries. Sending your entire corpus with every request drives up costs linearly, and models lose focus on relevant information in long contexts—a well-documented issue known as the "lost in the middle" effect (Liu et al., 2023). RAG retrieves only what's relevant, keeping costs and latency predictable.

**Q: How do I combine BM25 and vector search without running two separate systems?**

Use a vector database that supports both dense and sparse vectors in the same collection. Milvus 2.6 stores both vector types per document and returns fused results from a single query. You adjust the balance between keyword and semantic matching by changing a weight parameter—no separate indexes, no result merging, no sync headaches.

**Q: What's the first thing I should add to improve my existing RAG pipeline?**

Query routing. It's the highest-impact, lowest-effort improvement. Most production systems see a significant share of queries that don't need retrieval at all—common-sense questions, simple calculations, general knowledge. Routing these directly to the LLM cuts unnecessary retrieval calls and improves response time immediately.

**Q: How do I figure out which stage of my RAG pipeline is causing bad results?**

Evaluate each stage independently. Use F1 score for routing accuracy, Recall@K and NDCG@10 for retrieval quality, Precision@3 for reranking, and faithfulness metrics for generation. Set baselines from offline test data and monitor each stage in production. When answer quality drops, you can trace it to the specific stage that regressed instead of guessing.

Build Smarter RAG with Routing and Hybrid Retrieval


Beyond Naive RAG: Build Smarter Systems with Query Routing and Hybrid Retrieval


Data Addressing Deep Dive: From HashMap to Milvus


A Deep Dive into Data Addressing in Storage Systems: From HashMap to HDFS, Kafka, Milvus, and Iceberg


GEO at Scale: Rank in AI Search Without AI Content Spam


GEO Content at Scale: How to Rank in AI Search Without Poisoning Your Brand



Vector search ranks results by embedding similarity — the closer the vectors, the higher the result. Some systems add a model-based reranker (BGE, Voyage, Cohere) to improve ordering. But neither approach handles a fundamental requirement in production: **business context matters as much as semantic relevance, sometimes more.**

An e-commerce site needs to surface in-stock products from official stores first. A content platform wants to pin recent announcements. An enterprise knowledge base needs authoritative documents at the top. When ranking relies solely on vector distance, these rules get ignored. The results may be relevant, but they're not appropriate.

**[Boost Ranker](https://milvus.io/docs/reranking.md)**, introduced in [Milvus](https://milvus.io/intro) 2.6, solves this. It lets you adjust search result rankings using metadata rules — no index rebuild, no model change. This article covers how it works, when to use it, and how to implement it with code.

## What Is Boost Ranker?

**Boost Ranker is a lightweight, rule-based reranking feature in Milvus 2.6.2** that adjusts [vector search](https://zilliz.com/learn/vector-similarity-search) results using scalar metadata fields. Unlike model-based rerankers that call external LLMs or embedding services, Boost Ranker operates entirely within Milvus using simple filter-and-weight rules. No external dependencies, minimal latency overhead — suitable for real-time use.

You configure it through the [Function API](https://milvus.io/docs/manage-functions.md). After vector search returns a set of candidates, Boost Ranker applies three operations:

1. **Filter:** identify results matching specific conditions (e.g., `is_official == true`)
2. **Boost:** multiply their scores by a configured weight
3. **Shuffle:** optionally add a small random factor (0–1) to introduce diversity

### How It Works Under the Hood

Boost Ranker runs inside Milvus as a post-processing step:

1. **Vector search** — each segment returns candidates with IDs, similarity scores, and metadata.
2. **Apply rules** — the system filters matching records and adjusts their scores using the configured weight and optional `random_score`.
3. **Merge and sort** — all candidates are combined and re-sorted by updated scores to produce the final Top-K results.

Because Boost Ranker only operates on already-retrieved candidates — not the full dataset — the additional computational cost is negligible.

## When Should You Use Boost Ranker?

### Boosting Important Results

The most common use case: layer simple business rules on top of semantic search.

- **E-commerce:** boost products from flagship stores, official sellers, or paid promotions. Push items with high recent sales or click-through rates higher.
- **Content platforms:** prioritize recently published content via a `publish_time` field, or boost posts from verified accounts.
- **Enterprise search:** give higher priority to documents where `doctype == "policy"` or `is_canonical == true`.

All configurable with a filter + weight. No embedding model changes, no index rebuilds.

### Demoting Without Removing

Boost Ranker can also lower ranking for certain results — a softer alternative to hard filtering.

- **Low-stock products:** if `stock < 10`, reduce their weight slightly. Still findable, but won't dominate top positions.
- **Sensitive content:** lower the weight of flagged content instead of removing it entirely. Limits exposure without hard censorship.
- **Stale documents:** documents where `year < 2020` get ranked lower so newer content surfaces first.

Users can still find demoted results by scrolling or searching more precisely, but they won't crowd out more relevant content.

### Adding Diversity with Controlled Randomness

When many results have similar scores, the Top-K can look identical across queries. Boost Ranker's `random_score` parameter introduces slight variation:

```json
"random_score": {
  "seed": 126,
  "field": "id"
}
```

- `seed`: controls overall randomness for reproducibility
- `field`: usually the primary key `id`, ensures the same record gets the same random value each time

This is useful for **diversifying recommendations** (preventing the same items from always appearing first) and **exploration** (combining fixed business weights with small random perturbations).

### Combining Boost Ranker with Other Rankers

Boost Ranker is set via the Function API with `params.reranker = "boost"`. Two things to know about combining it:

- **Limitation:** in hybrid (multi-vector) search, Boost Ranker cannot be the top-level ranker. But it can be used inside each individual `AnnSearchRequest` to adjust results before they're merged.
- **Common combinations:**
  - **RRF + Boost:** use RRF to merge multi-modal results, then apply Boost for metadata-based fine-tuning.
  - **Model ranker + Boost:** use a model-based ranker for semantic quality, then Boost for business rules.

## How to Configure Boost Ranker

Boost Ranker is configured through the Function API. For more complex logic, combine it with `FunctionScore` to apply multiple rules together.

### Required Fields

When creating a `Function` object:

- `name`: any custom name
- `input_field_names`: must be an empty list `[]`
- `function_type`: set to `FunctionType.RERANK`
- `params.reranker`: must be `"boost"`

### Key Parameters

**`params.weight` (required)**

The multiplier applied to matching records' scores. How you set it depends on the metric:

| Metric Type | To Boost Results | To Demote Results |
|-------------|-----------------|-------------------|
| Higher-is-better (COSINE, IP) | `weight > 1` | `weight < 1` |
| Lower-is-better (L2/Euclidean) | `weight < 1` | `weight > 1` |

**`params.filter` (optional)**

A condition that selects which records get their scores adjusted:

- `"doctype == 'abstract'"`
- `"is_premium == true"`
- `"views > 1000 and category == 'tech'"`

Only matching records are affected. Everything else keeps its original score.

**`params.random_score` (optional)**

Adds a random value between 0 and 1 for diversity. See the randomness section above for details.

### Single vs. Multiple Rules

**Single rule** — when you have one business constraint (e.g., boost official docs), pass the ranker directly to `search(..., ranker=ranker)`.

**Multiple rules** — when you need several constraints (prioritize in-stock items + demote low-rated products + add randomness), create multiple `Function` objects and combine them with `FunctionScore`. You configure:

- `boost_mode`: how each rule combines with the original score (`multiply` or `add`)
- `function_mode`: how multiple rules combine with each other (`multiply` or `add`)

## Hands-On: Prioritizing Official Documents

Let's walk through a concrete example: making official documents rank higher in a document search system.

### Schema

A collection called `milvus_collection` with these fields:

| Field | Type | Purpose |
|-------|------|---------|
| `id` | INT64 | Primary key |
| `content` | VARCHAR | Document text |
| `embedding` | FLOAT_VECTOR (3072) | Semantic vector |
| `source` | VARCHAR | Origin: "official", "community", or "ticket" |
| `is_official` | BOOL | `True` if `source == "official"` |

The `source` and `is_official` fields are the metadata Boost Ranker will use to adjust rankings.

### Setup Code

```python
from pymilvus import (
    MilvusClient,
    DataType,
    Function,
    FunctionType,
)

# 1. Connect to Milvus
client = MilvusClient(uri="http://localhost:19530")
collection_name = "milvus_collection"

# If it already exists, drop it first for repeated testing
if collection_name in client.list_collections():
    client.drop_collection(collection_name)

# 2. Define schema
schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=False,
)

schema.add_field(
    field_name="id",
    datatype=DataType.INT64,
    is_primary=True,
)
schema.add_field(
    field_name="content",
    datatype=DataType.VARCHAR,
    max_length=512,
)
schema.add_field(
    field_name="source",
    datatype=DataType.VARCHAR,
    max_length=32,
)
schema.add_field(
    field_name="is_official",
    datatype=DataType.BOOL,
)
schema.add_field(
    field_name="embedding",
    datatype=DataType.FLOAT_VECTOR,
    dim=3072,
)

text_embedding_function = Function(
    name="openai_embedding",
    function_type=FunctionType.TEXTEMBEDDING,
    input_field_names=["content"],
    output_field_names=["embedding"],
    params={
        "provider": "openai",
        "model_name": "text-embedding-3-large"
    }
)

schema.add_function(text_embedding_function)

# 3. Create Collection
client.create_collection(
    collection_name=collection_name,
    schema=schema,
)

# 4. Create index
index_params = client.prepare_index_params()

index_params.add_index(
    field_name="embedding",
    index_type="IVF_FLAT",
    metric_type="COSINE",
    params={"nlist": 16},
)

client.create_index(
    collection_name=collection_name,
    index_params=index_params,
)

# 5. Load Collection into memory
client.load_collection(collection_name=collection_name)

docs = [
    {
        "id": 1,
        "content": "How to deploy Milvus on Kubernetes (Official Guide)",
        "source": "official",
        "is_official": True
    },
    {
        "id": 2,
        "content": "Quick deployment of Milvus with Docker Compose (Official Tutorial)",
        "source": "official",
        "is_official": True
    },
    {
        "id": 3,
        "content": "Community experience: Lessons learned from deploying Milvus",
        "source": "community",
        "is_official": False
    },
    {
        "id": 4,
        "content": "Ticket record: Milvus deployment issue",
        "source": "ticket",
        "is_official": False
    },
]

client.insert(
    collection_name=collection_name,
    data=docs,
)
```

### Comparing Results: With and Without Boost Ranker

First, run a baseline search without Boost Ranker. Then add Boost Ranker with `filter: is_official == true` and `weight: 1.2`, and compare.

```python
# 6. Baseline search (without Boost Ranker)
query_vector = "how to deploy milvus"

search_params = {
    "metric_type": "COSINE",
    "params": {"nprobe": 2},
}

results = client.search(
    collection_name=collection_name,
    data=[query_vector],
    anns_field="embedding",
    search_params=search_params,
    limit=4,
    output_fields=["content", "source", "is_official"],
)

print("=== Baseline search (no Boost Ranker) ===")
for hit in results[0]:
    entity = hit["entity"]
    print(
        f"id={hit['id']}, "
        f"score={hit['distance']:.4f}, "
        f"source={entity['source']}, "
        f"is_official={entity['is_official']}"
    )

# 7. Define Boost Ranker: apply weight to documents where is_official == true
boost_official_ranker = Function(
    name="boost_official",
    input_field_names=[],               # Boost Ranker requires this to be an empty list
    function_type=FunctionType.RERANK,
    params={
        "reranker": "boost",            # Specify Boost Ranker
        "filter": "is_official==true",
        # For COSINE / IP (higher score is better), use weight > 1 to boost
        "weight": 1.2
    },
)

boosted_results = client.search(
    collection_name=collection_name,
    data=[query_vector],
    anns_field="embedding",
    search_params=search_params,
    limit=4,
    output_fields=["content", "source", "is_official"],
    ranker=boost_official_ranker,
)

print("\n=== Search with Boost Ranker (official boosted) ===")
for hit in boosted_results[0]:
    entity = hit["entity"]
    print(
        f"id={hit['id']}, "
        f"score={hit['distance']:.4f}, "
        f"source={entity['source']}, "
        f"is_official={entity['is_official']}"
    )
```

### Results

```
=== Baseline search (no Boost Ranker) ===
id=1, score=0.7351, source=official, is_official=True
id=4, score=0.7017, source=ticket, is_official=False
id=3, score=0.6706, source=community, is_official=False
id=2, score=0.6435, source=official, is_official=True

=== Search with Boost Ranker (official boosted) ===
id=1, score=0.8821, source=official, is_official=True
id=2, score=0.7722, source=official, is_official=True
id=4, score=0.7017, source=ticket, is_official=False
id=3, score=0.6706, source=community, is_official=False
```

The key change: document `id=2` (official) jumped from 4th to 2nd place because its score was multiplied by 1.2. Community posts and ticket records aren't removed — they just rank lower. That's the point of Boost Ranker: keep semantic search as the foundation, then layer business rules on top.

## Conclusion

[Boost Ranker](https://milvus.io/docs/reranking.md) gives you a way to inject business logic into vector search results without touching your embeddings or rebuilding indexes. Boost official content, demote stale results, add controlled diversity — all through simple filter + weight configuration in the [Milvus Function API](https://milvus.io/docs/manage-functions.md).

Whether you're building RAG pipelines, recommendation systems, or enterprise search, Boost Ranker helps bridge the gap between what's semantically similar and what's actually useful to your users.

If you're working on search ranking and want to discuss your use case:

- Join the [Milvus Slack community](https://slack.milvus.io/) to connect with other developers building search and retrieval systems.
- [Book a free 20-minute Milvus Office Hours session](https://milvus.io/office-hours) to walk through your ranking logic with the team.
- If you'd rather skip infrastructure setup, [Zilliz Cloud](https://cloud.zilliz.com/signup) (managed Milvus) has a free tier to get started.

---

A few questions that come up when teams start using Boost Ranker:

**Can Boost Ranker replace a model-based reranker like Cohere or BGE?**
They solve different problems. Model-based rerankers re-score results by semantic quality — they're good at deciding "which document actually answers the question." Boost Ranker adjusts scores by business rules — it decides "which relevant document should appear first." In practice, you often want both: a model ranker for semantic relevance, then Boost Ranker for business logic on top.

**Does Boost Ranker add significant latency?**
No. It operates on the already-retrieved candidate set (typically the Top-K from vector search), not the full dataset. The operations are simple filter-and-multiply, so the overhead is negligible compared to the vector search itself.

**How do I set the weight value?**
Start with small adjustments. For COSINE similarity (higher is better), a weight of 1.1–1.3 is usually enough to noticeably shift rankings without overriding semantic relevance entirely. Test with your actual data — if boosted results with low similarity start dominating, lower the weight.

**Can I combine multiple Boost Ranker rules?**
Yes. Create multiple `Function` objects and combine them using `FunctionScore`. You control how rules interact through `boost_mode` (how each rule combines with the original score) and `function_mode` (how rules combine with each other) — both support `multiply` and `add`.

Milvus Boost Ranker: Add Business Rules to Vector Search


How to Use Milvus Boost Ranker for Business-Aware Vector Search


Fix RAG Retrieval Errors with CRAG, LangGraph, and Milvus




Your RAG prototype worked great. Then it went to production, traffic grew, and now your vector database bill has gone from \$500 to \$5,000 a month. Sound familiar?

This is one of the most common scaling problems in AI applications right now. You've built something that creates real value, but the infrastructure costs are growing faster than your user base is growing. And when you look at the bill, the vector database is often the biggest surprise — in the deployments we've seen, it can account for roughly 40-50% of total application cost, second only to LLM API calls.

In this guide, I'll walk through where the money actually goes and the specific things you can do to bring it down — in many cases by 60-80%. I'll use [Milvus](https://milvus.io/), the most popular open-source vector database, as the primary example since that's what I know best, but the principles apply to most vector databases.

*To be clear:* *[Milvus](https://milvus.io/)* *itself is free and open source — you never pay for the software. The cost comes entirely from the infrastructure you run it on: cloud instances, memory, storage, and network. The good news is that most of that infrastructure cost is reducible.*

## Where Does the Money Actually Go When Using a VectorDB?

Let's start with a concrete example. Say you have 100 million vectors, 768 dimensions, stored as float32 — a pretty typical RAG setup. Here's roughly what that costs on AWS per month:

| **Cost Component** | **Share** | **~Monthly Cost** | **Notes** |
| --- | --- | --- | --- |
| Compute   (CPU + memory) | 85-90% | \$2,800 | The big one — mostly driven by memory |
| Network | 5-10% | \$250 | Cross-AZ traffic, large result payloads |
| Storage | 2-5% | \$100 | Cheap — object storage (S3/MinIO) is ~\$0.03/GB |

The takeaway is simple: memory is where 85-90% of your money goes. Network and storage matter at the margins, but if you want to cut costs meaningfully, memory is the lever. Everything in this guide focuses on that.

**Quick note on network and storage:** You can reduce network costs by only returning the fields you need (ID, score, key metadata) and avoiding cross-region queries. For storage, Milvus already separates storage from compute — your vectors sit in cheap object storage like S3, so even at 100M vectors, storage is usually under \$50/month. Neither of these will move the needle like memory optimization will.

## Why Memory Is So Expensive for Vector Search

If you're coming from traditional databases, the memory requirements for vector search can be surprising. A relational database can leverage disk-based B-tree indexes and the OS page cache. Vector search is different — it involves massive floating-point computation, and indexes like HNSW or IVF need to stay loaded in memory to deliver millisecond-level latency.

Here's a quick formula to estimate your memory needs:

**Memory required = (vectors × dimensions × 4 bytes) × index multiplier**

For our 100M × 768 × float32 example with HNSW (multiplier ~1.8x):

-   Raw data: 100M × 768 × 4 bytes ≈ 307 GB
-   With HNSW index: 307 GB × 1.8 ≈ 553 GB
-   With OS overhead, cache, and headroom: ~768 GB total
-   On AWS: 3× r6i.8xlarge (256 GB each) ≈ \$2,800/month

**That's the baseline. Now let's look at how to bring it down.**

## 1. Pick the Right Index to Get 4x Less Memory Usage

This is the single highest-impact change you can make. For the same 100M-vector dataset, memory usage can vary by 4-6x depending on your index choice.

-   **FLAT / IVF_FLAT**: almost no compression, so memory usage stays close to the raw data size, around **300 GB**
-   **HNSW**: stores an extra graph structure, so memory usage is usually **1.5x to 2.0x** the raw data size, or about **450 to 600 GB**
-   **IVF_SQ8**: compresses float32 values into uint8, giving about **4x compression**, so memory use can drop to around **75 to 100 GB**
-   **IVF_PQ / DiskANN**: use stronger compression or a disk-based index, so memory can drop further to about **30 to 60 GB**

Many teams start with HNSW because it has the best query speed, but they end up paying 3-5x more than they need to.

Here's how the main index types compare:

| **Index** | **Memory Multiplier** | **Query Speed** | **Recall** | **Best For** |
| --- | --- | --- | --- | --- |
| FLAT | ~1.0x | Slow | 100% | Small datasets (<1M), testing |
| IVF_FLAT | ~1.05x | Medium | 95-99% | General use |
| IVF_SQ8 | ~0.30x | Medium | 93-97% | Cost-sensitive production (recommended) |
| IVF_PQ | ~0.12x | Fast | 70-80% | Very large datasets, coarse retrieval |
| HNSW | ~1.8x | Very fast | 98-99% | Only when latency matters more than cost |
| DiskANN | ~0.08x | Medium | 95-98% | Very large scale with NVMe SSDs |

**The bottom line:** Switching from HNSW or IVF_FLAT to IVF_SQ8 typically drops recall by only 2-3% (e.g., from 97% to 94-95%) while cutting memory cost by about 70%. For most RAG workloads, that tradeoff is absolutely worth it. If you're doing coarse retrieval or your accuracy bar is lower, IVF_PQ or IVF_RABITQ can further boost savings.

**My recommendation:** If you're running HNSW in production and cost is a concern, try IVF_SQ8 on a test collection first. Measure recall on your actual queries. Most teams are surprised by how small the accuracy drop is.

## 2. Stop Loading Everything into Memory for 60%-80% Cost Reduction

Even after picking a more efficient index, you might still have more data in memory than necessary. Milvus offers two ways to fix this: **MMap (available since 2.3) and tiered storage (available since 2.6). Both can reduce memory usage by 60-80%.**

The core idea behind both is the same: not all your data needs to live in memory at all times. The difference is how they handle the data that's not in memory.

### MMap (Memory-Mapped Files)

MMap maps your data files from local disk into the process address space. The full dataset remains on the node's local disk, and the OS loads pages into memory on demand—only when they're accessed. Before using MMap, all data gets downloaded from object storage (S3/MinIO) to the QueryNode's local disk.

-   Memory usage drops to ~10-30% of full-load mode
-   Latency stays stable and predictable (data is on local disk, no network fetch)
-   Tradeoff: local disk must be large enough to hold the full dataset

### Tiered Storage

Tiered storage takes it a step further. Instead of downloading everything to the local disk, it uses the local disk as a cache for hot data and keeps object storage as the primary layer. Data is fetched from object storage only when needed.

-   Memory usage drops to <10% of full-load mode
-   Local disk usage also drops — only hot data is cached (usually 10-30% of total)
-   Tradeoff: cache misses add 50-200ms latency (fetching from object storage)

### Data flow and resource usage

| **Mode** | **Data Flow** | **Memory Usage** | **Local Disk Usage** | **Latency** |
| --- | --- | --- | --- | --- |
| Traditional full load | Object storage → memory (100%) | Very high (100%) | Low (temporary only) | Very low and stable |
| MMap | Object storage → local disk (100%) → memory (on demand) | Low (10-30%) | High (100%) | Low and stable |
| Tiered storage | Object storage ↔ local cache (hot data) → memory (on demand) | Very low (<10%) | Low (hot data only) | Low on cache hit, higher on cache miss |

**Hardware recommendation:** both methods depend heavily on local disk I/O, so **NVMe SSDs** are strongly recommended, ideally with **IOPS above 10,000**.

### MMap vs. Tiered Storage: Which One Should You Use?

| **Your Situation** | **Use This** | **Why** |
| --- | --- | --- |
| Latency-sensitive (P99 < 20ms) | MMap | Data is already on local disk — no network fetch, stable latency |
| Uniform access (no clear hot/cold split) | MMap | Tiered storage needs hot/cold skew to be effective; without it, cache hit rate is low |
| Cost is the priority (occasional latency spikes OK) | Tiered storage | Saves on both memory and local disk (70-90% less disk) |
| Clear hot/cold pattern (80/20 rule) | Tiered storage | Hot data stays cached, cold data stays cheap in object storage |
| Very large scale (>500M vectors) | Tiered storage | One node's local disk often can't hold the full dataset at this scale |

**Note:** MMap requires Milvus 2.3+. Tiered storage requires Milvus 2.6+. Both work best with NVMe SSDs (10,000+ IOPS recommended).

### How to Configure MMap

**Option 1: YAML configuration (recommended for new deployments)**

Edit the Milvus configuration file milvus.yaml and add the following settings under the queryNode section:

```
queryNode:
  mmap:
    vectorField: true      # vector data
    vectorIndex: true      # vector index (largest source of savings!)
    scalarField: true      # scalar data (recommended for RAG workloads)
    scalarIndex: true      # scalar index
    growingMmapEnabled: false  # incremental data stays in memory
```

**Option 2: Python SDK configuration (for existing collections)**

```
from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")

# You must release the collection before changing the mmap setting
client.release_collection("my_collection")

# Enable MMap
client.alter_collection_properties(
    collection_name="my_collection",
    properties={"mmap.enabled": True}
)

# Load the collection again to apply the MMap setting
client.load_collection("my_collection")

# Verify that the setting has taken effect
print(client.describe_collection("my_collection")["properties"])
# Output: {'mmap.enabled': 'True'}
```

### How to Configure Tiered Storage (Milvus 2.6+)

Edit the Milvus configuration file milvus.yaml and add the following settings under the queryNode section:

```
queryNode:
  segcore:
    tieredStorage:
      warmup:                                                                                                                                                      
          # Options: sync, async, disable                      
          # Specifies when tiered storage cache warm-up happens.                                                                                                                             
          # - "sync": data is loaded into the cache before the segment is considered fully loaded.                                                                                    
          # - "disable": data is not proactively loaded into the cache, and is loaded only when needed by Search/Query tasks.                                                                            
          # The default is "sync", but vector fields default to "disable".                                                                                                            
          scalarField: sync                                                                                                                                          
          scalarIndex: sync                                                                                                                                          
          vectorField: disable # Cache warm-up for raw vector field data is disabled by default.
          vectorIndex: sync
      memoryHighWatermarkRatio: 0.85   # Start eviction when memory usage exceeds 85%
      memoryLowWatermarkRatio: 0.70    # Stop eviction when memory usage drops to 70%
      diskHighWatermarkRatio: 0.80     # High watermark for disk eviction
      diskLowWatermarkRatio: 0.75      # Low watermark for disk eviction
      evictionEnabled: true            # Must be enabled!
      backgroundEvictionEnabled: true  # Background eviction thread
      cacheTtl: 3600                   # Automatically evict if not accessed for 1 hour
```

## Use Lower-Dimensional Embeddings

This one is easy to overlook, but the dimension directly scales your cost. Memory, storage, and compute all grow linearly with dimension count. A 1536-dim model costs roughly 4x more infrastructure than a 384-dim model for the same data.

Query cost scales the same way — cosine similarity is O(D), so 768-dim vectors take about twice the compute of 384-dim vectors per query. In high-QPS workloads, that difference translates directly into fewer nodes needed.

Here's how common embedding models compare (using 384-dim as the 1.0x baseline):

| **Model** | **Dimensions** | **Relative Cost** | **Recall** | **Best For** |
| --- | --- | --- | --- | --- |
| text-embedding-3-large | 3072 | 8.0x | 98%+ | When accuracy is non-negotiable (research, healthcare) |
| text-embedding-3-small | 1536 | 4.0x | 95-97% | General RAG workloads |
| DistilBERT | 768 | 2.0x | 92-95% | Good cost-performance balance |
| all-MiniLM-L6-v2 | 384 | 1.0x | 88-92% | Cost-sensitive workloads |

**Practical advice:** Don't assume you need the biggest model. Test on a representative sample of your actual queries (1M vectors is usually enough) and find the lowest-dimension model that meets your accuracy bar. Many teams discover that 768 dimensions works just as well as 1536 for their use case.

**Already committed to a high-dimensional model?** You can reduce dimensions after the fact. PCA (Principal Component Analysis) can strip out redundant features, and [Matryoshka embeddings](https://milvus.io/blog/matryoshka-embeddings-detail-at-multiple-scales.md) let you truncate to the first N dimensions while retaining most of the quality. Both are worth trying before re-embedding your entire dataset.

## Manage Data Lifecycle with Compaction and TTL

This one is less glamorous but still matters, especially for long-running production systems. Milvus uses an append-only storage model: when you delete data, it's marked as deleted but not removed immediately. Over time, this dead data accumulates, wastes storage space, and causes queries to scan more rows than they need to.

### Compaction: Reclaim Storage from Deleted Data

Compaction is Milvus's background process for cleaning up. It merges small segments, physically removes deleted data, and rewrites compacted files. You'll want this if:

-   You have frequent writes and deletes (product catalogs, content updates, real-time logs)
-   Your segment count keeps growing (this increases per-query overhead)
-   Storage usage is growing much faster than your actual valid data

**Heads up:** Compaction is I/O-intensive. Schedule it during low-traffic periods (e.g., nightly) or tune the triggers carefully so it doesn't compete with production queries.

### TTL(Time to Live): Automatically Expire Old Vector Data

For data that naturally expires, TTL is cleaner than manual deletion. Set a lifetime on your data, and Milvus automatically marks it for deletion when it expires. Compaction handles the actual cleanup.

This is useful for:

-   Logs and session data — keep only the last 7 or 30 days
-   Time-sensitive RAG — prefer recent knowledge, let old documents expire
-   Real-time recommendations — only retrieve from recent user behavior

Together, compaction and TTL keep your system from silently accumulating waste. It's not the biggest cost lever, but it prevents the kind of slow storage creep that catches teams off guard.

## One More Option: Zilliz Cloud (Fully Managed Milvus)

Full disclosure: [Zilliz Cloud](https://zilliz.com/) is built by the same team behind Milvus, so take this with the appropriate grain of salt.

That said, here's the counterintuitive part: even though Milvus is free and open source, a managed service can actually cost less than self-hosting. The reason is simple — the software is free, but the cloud infrastructure to run it isn't, and you need engineers to operate and maintain it. If a managed service can do the same work with fewer machines and fewer engineer hours, your total bill goes down even after paying for the service itself.

[Zilliz Cloud](https://zilliz.com/) is a fully managed service built on Milvus and API-compatible with it. Two things are relevant to cost:

-   **Better performance per node.** Zilliz Cloud runs on Cardinal, our optimized search engine. Based on [VectorDBBench results](https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch), it delivers 3-5x higher throughput than open-source Milvus and is 10x faster. In practice, that means you need roughly one-third to one-fifth as many compute nodes for the same workload.
-   **Built-in optimizations.** The features covered in this guide — MMap, tiered storage, and index quantization — are built in and automatically tuned. Auto-scaling adjusts capacity based on actual load, so you're not paying for headroom you don't need.

![](https://assets.zilliz.com/Blog_Howto_Cut_Vector_Database_Costsby_Upto80_A_Pract_1_5230ab94bf.png)

[Migration](https://zilliz.com/zilliz-migration-service) is straightforward since the APIs and data formats are compatible. Zilliz also provides migration tooling to help. For a detailed comparison, see: [Zilliz Cloud vs. Milvus](https://zilliz.com/zilliz-vs-milvus)

## Summary: A Step-by-Step Plan to Cut Vector Database Costs

**If you only do one thing, do this: check your index type.**

If you're running HNSW on a cost-sensitive workload, switch to IVF_SQ8. That alone can cut memory cost by ~70% with minimal recall loss.

If you want to go further, here's the priority order:

-   **Switch your index** — HNSW → IVF_SQ8 for most workloads. Biggest bang for zero architectural change.
-   **Enable MMap or tiered storage** — Stop keeping everything in memory. This is a config change, not a redesign.
-   **Evaluate your embedding dimensions** — Test whether a smaller model meets your accuracy needs. This requires re-embedding but the savings compound.
-   **Set up compaction and TTL** — Prevent silent data bloat, especially if you have frequent writes/deletes.

Combined, these strategies can reduce your vector database bill by 60-80%. Not every team needs all four — start with the index change, measure the impact, and work your way down the list.

For teams looking to reduce operational work and improve cost efficiency, [Zilliz Cloud](https://zilliz.com/) (managed Milvus) is another option.

If you're working through any of these optimizations and want to compare notes, the [Milvus community Slack](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) is a good place to ask questions. You can also join [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) for a quick chat with the engineering team about your specific setup.


Milvus Cost Optimization Guide: Cut Vector Database Costs by Up to 80%


How to Cut Vector Database Costs by Up to 80%: A Practical Milvus Optimization Guide




A query like “find romantic restaurants within 3 km” sounds simple. It's not, because it combines location filtering and semantic search. Most systems need to split this query across two databases, which means syncing data, merging results in code, and extra latency.

[Milvus](https://milvus.io) 2.6.4 eliminates this split. With a native **GEOMETRY** data type and an **R-Tree** index, Milvus can apply location and semantic constraints together in a single query. This makes hybrid spatial and semantic search much easier and more efficient.

This article explains why this change was needed, how GEOMETRY and R-Tree work inside Milvus, what performance gains to expect, and how to set it up with the Python SDK.

## The Limitations of Traditional Geo and Semantic Search

Queries like “romantic restaurants within 3 km”  are hard to handle for two reasons:

-   **"Romantic" needs semantic search.** The system has to vectorize restaurant reviews and tags, then find matches by similarity in embedding space. This only works in a vector database.
-   **"Within 3 km" needs spatial filtering.** Results must be restricted to “within 3 km of the user,” or sometimes “inside a specific delivery polygon or administrative boundary.”

In a traditional architecture, meeting both needs usually meant running two systems side by side:

-   **PostGIS / Elasticsearch** for geofencing, distance calculations, and spatial filtering.
-   A **vector database** for approximate nearest neighbor (ANN) search over embeddings.

This “two-database” design creates three practical problems:

-   **Painful data synchronization.** If a restaurant changes its address, you must update both the geo system and the vector database. Missing one update produces inconsistent results.
-   **Higher latency.** The application has to call two systems and merge their outputs, adding network round-trips and processing time.
-   **Inefficient filtering.** If the system ran vector search first, it often returned many results that were far from the user and had to be discarded later. If it applied location filtering first, the remaining set was still large, so the vector search step was still expensive.

Milvus 2.6.4 solves this by adding spatial geometry support directly to the vector database. Semantic search and location filtering now run in the same query. With everything in one system, hybrid search is faster and easier to manage.

## What GEOMETRY Adds to Milvus

Milvus 2.6 introduces a scalar field type called DataType.GEOMETRY. Instead of storing locations as separate longitude and latitude numbers, Milvus now stores geometric objects: points, lines, and polygons. Queries like "is this point inside a region?" or "is it within X meters?" become native operations. There’s no need to build workarounds over raw coordinates.

The implementation follows the [](https://www.ogc.org/standard/sfa/)**OpenGIS Simple Features Access standard**, so it works with most existing geospatial tooling. Geometry data is stored and queried using **WKT (Well-Known Text)**, a standard text format that is readable by humans and parseable by programs.

Supported geometry types:

-   **POINT**: a single location, such as a store address or a vehicle’s real-time position
-   **LINESTRING**: a line, such as a road centerline or a movement path
-   **POLYGON**: an area, such as an administrative boundary or a geofence
-   **Collection types**: MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION

It also supports standard spatial operators, including:

-   **Spatial relationships**: containment (ST_CONTAINS, ST_WITHIN), intersection (ST_INTERSECTS, ST_CROSSES), and contact (ST_TOUCHES)
-   **Distance operations**: computing distances between geometries (ST_DISTANCE) and filtering objects within a given distance (ST_DWITHIN)

## How R-Tree Indexing Works Inside Milvus

GEOMETRY support is built into the Milvus query engine, not just exposed as an API feature. ISpatial data is indexed and processed directly inside the engine using the R-Tree (Rectangle Tree) index.

An **R-Tree** groups nearby objects using **minimum bounding rectangles (MBRs)**. During a query, the engine skips large regions that do not overlap with the query geometry and only runs detailed checks on a small set of candidates. This is much faster than scanning every object.

### How Milvus Builds the R-Tree

R-Tree construction happens in layers:

| **Level** | **What Milvus Does** | **Intuitive Analogy** |
| --- | --- | --- |
| **Leaf level** | For each geometry object (point, line, or polygon), Milvus computes its minimum bounding rectangle (MBR) and stores it as a leaf node. | Wrapping each item in a transparent box that fits it exactly. |
| **Intermediate levels** | Nearby leaf nodes are grouped together (typically 50–100 at a time), and a larger parent MBR is created to cover all of them. | Putting packages from the same neighborhood into a single delivery crate. |
| **Root level** | This grouping continues upward until all data is covered by a single root MBR. | Loading all crates onto one long-haul truck. |

With this structure in place, spatial query complexity drops from a full scan **O(n)** to **O(log n)**. In practice, queries over millions of records can go from hundreds of milliseconds down to just a few milliseconds, without losing accuracy.

### How Queries are Executed: Two-Phase Filtering

To balance speed and correctness, Milvus uses a **two-phase filtering** strategy:

-   **Rough filter:** the R-Tree index first checks whether the query’s bounding rectangle overlaps with other bounding rectangles in the index. This quickly removes most unrelated data and keeps only a small set of candidates. Because these rectangles are simple shapes, the check is very fast, but it can include some results that don’t actually match.
-   **Fine filter**: the remaining candidates are then checked using **GEOS**, the same geometry library used by systems like PostGIS. GEOS runs exact geometry calculations, such as whether shapes intersect or one contains another, to produce correct final results.

![](https://assets.zilliz.com/blog_Hybrid_Spatialand_Vector_Searchwith_Milvus2_6_1_978d62cb30.png)

Milvus accepts geometry data in **WKT (Well-Known Text)** format but stores it internally as **WKB (Well-Known Binary).** WKB is more compact, which cuts storage and improves I/O. GEOMETRY fields also support memory-mapped (mmap) storage, so large spatial datasets don't need to fit entirely in RAM.

## Performance Improvements with R-Tree

### Query Latency Stays Flat as Data Grows.

Without an R-Tree index, query time scales linearly with data size — 10x more data means roughly 10x slower queries.

With R-Tree, query time grows logarithmically. On datasets with millions of records, spatial filtering can be tens to hundreds of times faster than a full scan.

### Accuracy is Not Sacrificed For Speed

The R-Tree narrows candidates by bounding box, then GEOS checks each one with exact geometry math. Anything that looks like a match but actually falls outside the query area gets removed in the second pass.

### Hybrid Search Throughput Improves

![](https://assets.zilliz.com/blog_Hybrid_Spatialand_Vector_Searchwith_Milvus2_6_2_b458b24bf6.png)

The R-Tree removes records outside the target area first. Milvus then runs vector similarity (L2, IP, or cosine) only on the remaining candidates. Fewer candidates means lower search cost and higher queries per second (QPS).

## Getting Started: GEOMETRY with the Python SDK

### Define the Collection and Create Indexes

First, define a DataType.GEOMETRY field in the collection schema. This allows Milvus to store and query geometric data.

```  
from pymilvus import MilvusClient, DataType  
import numpy as np  
# Connect to Milvus  
milvus_client = MilvusClient("[http://localhost:19530](http://localhost:19530)")  
collection_name = "lb_service_demo"  
dim = 128  
# 1. Define schema  
schema = milvus_client.create_schema(enable_dynamic_field=True)  
schema.add_field("id", DataType.INT64, is_primary=True)  
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=dim)  
schema.add_field("location", DataType.GEOMETRY)  # Define geometry field  
schema.add_field("poi_name", DataType.VARCHAR, max_length=128)  
# 2. Create index parameters  
index_params = milvus_client.prepare_index_params()  
# Create an index for the vector field (e.g., IVF_FLAT)  
index_params.add_index(  
   field_name="vector",  
   index_type="IVF_FLAT",  
   metric_type="L2",  
   params={"nlist": 128}  
)  
# Create an R-Tree index for the geometry field (key step)  
index_params.add_index(  
   field_name="location",  
   index_type="RTREE"  # Specify the index type as RTREE  
)  
# 3. Create collection  
if milvus_client.has_collection(collection_name):  
   milvus_client.drop_collection(collection_name)  
milvus_client.create_collection(  
   collection_name=collection_name,  
   schema=schema,  
   index_params=index_params,  # Create the collection with indexes attached  
   consistency_level="Strong"  
)  
print(f"Collection {collection_name} created with R-Tree index.")  
```

### Insert Data

When inserting data, geometry values must be in WKT (Well-Known Text) format. Each record includes the geometry, the vector, and other fields.

```  
# Mock data: random POIs in a region of Beijing  
data = []  
# Example WKT: POINT(longitude latitude)  
geo_points = [  
   "POINT(116.4074 39.9042)",  # Near the Forbidden City  
   "POINT(116.4600 39.9140)",  # Near Guomao  
   "POINT(116.3200 39.9900)",  # Near Tsinghua University  
]  
for i, wkt in enumerate(geo_points):  
   vec = np.random.random(dim).tolist()  
   data.append({  
       "id": i,  
       "vector": vec,  
       "location": wkt,  
       "poi_name": f"POI_{i}"  
   })  
res = milvus_client.insert(collection_name=collection_name, data=data)  
print(f"Inserted {res['insert_count']} entities.")  
```

### Run a Hybrid Spatial-Vector Query (Example)

**Scenario:** find the top 3 POIs that are most similar in vector space and located within 2 kilometers of a given point, such as the user’s location.

Use the ST_DWITHIN operator to apply the distance filter. The distance value is specified in **meters.**

```  
# Load the collection into memory  
milvus_client.load_collection(collection_name)  
# User location (WKT)  
user_loc_wkt = "POINT(116.4070 39.9040)"  
search_vec = np.random.random(dim).tolist()  
# Build the filter expression: use ST_DWITHIN for a 2000-meter radius filter  
filter_expr = f"ST_DWITHIN(location, '{user_loc_wkt}', 2000)"  
# Execute the search  
search_res = milvus_client.search(  
   collection_name=collection_name,  
   data=[search_vec],  
   filter=filter_expr,  # Inject geometry filter  
   limit=3,  
   output_fields=["poi_name", "location"]  
)  
print("Search Results:")  
for hits in search_res:  
   for hit in hits:  
       print(f"ID: {hit['id']}, Score: {hit['distance']:.4f}, Name: {hit['entity']['poi_name']}")  
```

## Tips for Production Use

-   **Always create an R-Tree index on GEOMETRY fields.** For datasets above 10,000 entities, spatial filters without an RTREE index fall back to a full scan, and performance drops sharply.
-   **Use a consistent coordinate system.** All location data must use the same system (e.g., [](https://en.wikipedia.org/wiki/World_Geodetic_System)[WGS 84](https://en.wikipedia.org/wiki/World_Geodetic_System)). Mixing coordinate systems breaks distance and containment calculations.
-   **Pick the right spatial operator for the query.** ST_DWITHIN for "within X meters" searches. ST_CONTAINS or ST_WITHIN for geofencing and containment checks.
-   **NULL geometry values are handled automatically.** If the GEOMETRY field is nullable (nullable=True), Milvus skips NULL values during spatial queries. No extra filtering logic needed.

## Deployment Requirements

To use these features in production, make sure your environment meets the following requirements.

**1. Milvus Version**

You must run **Milvus 2.6.4 or later**. Earlier versions do not support DataType.GEOMETRY or the **RTREE** index type.

**2. SDK Versions**

-   **PyMilvus**: upgrade to the latest version (the **2.6.x** series is recommended). This is required for proper WKT serialization and for passing RTREE index parameters.
-   **Java / Go / Node SDKs**: check the release notes for each SDK and confirm that they are aligned with the **2.6.4** proto definitions.

**3. Built-in Geometry Libraries**

The Milvus server already includes Boost.Geometry and GEOS, so you don’t need to install these libraries yourself.

**4. Memory Usage and Capacity Planning**

R-Tree indexes use extra memory. When planning capacity, remember to budget for geometry indexes as well as vector indexes like HNSW or IVF. The GEOMETRY field supports memory-mapped (mmap) storage, which can reduce memory usage by keeping part of the data on disk.

## Conclusion

Location-based semantic search needs more than bolting a geo filter onto a vector query. It requires built-in spatial data types, proper indexes, and a query engine that can handle location and vectors together.

**Milvus 2.6.4** solves this with native **GEOMETRY** fields and **R-Tree** indexes. Spatial filtering and vector search run in a single query, against a single data store. The R-Tree handles fast spatial pruning while GEOS ensures exact results.

For applications that need location-aware retrieval, this removes the complexity of running and syncing two separate systems.

If you’re working on location-aware or hybrid spatial and vector search, we’d love to hear your experience.

**Have questions about Milvus?** Join our [Slack channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session.

Hybrid Spatial and Vector Search with Milvus 2.6.4 (Geometry & R-Tree)


How to Use Hybrid Spatial and Vector Search with Milvus 



*This post was originally published on* *[the Airtable Medium](https://medium.com/airtable-eng/productionizing-semantic-search-how-we-built-and-scaled-vector-infrastructure-at-airtable-180fff11a136)* *channel and is reposted here with permission.*

As semantic search at Airtable evolved from a concept into a core product feature, the Data Infrastructure team faced the challenge of scaling it. As detailed in our [previous post on Building the Embedding System](https://medium.com/airtable-eng/building-a-resilient-embedding-system-for-semantic-search-at-airtable-d5fdf27807e2), we had already designed a robust, eventually consistent application layer to handle the embedding lifecycle. But one critical piece was still missing from our architecture diagram: the vector database itself.

We needed a storage engine capable of indexing and serving billions of embeddings, supporting massive multi-tenancy, and maintaining performance and availability targets in a distributed cloud environment. This is the story of how we architected, hardened, and evolved our vector search platform to become a core pillar of Airtable’s infrastructure stack.

## Background

At Airtable, our goal is to help customers work with their data in powerful, intuitive ways. With the emergence of increasingly powerful and accurate LLMs, features that leverage the semantic meaning of your data have become core to our product.

## How We Use Semantic Search

### Omni (Airtable’s AI Chat) answering real questions from large datasets

Imagine asking a natural-language question of your base (database) with half a million rows, and getting a correct, context-rich answer. For example:

“What are customers saying about battery life lately?”

On small datasets, it’s possible to send all rows directly to an LLM. At scale, that quickly becomes infeasible. Instead, we needed a system capable of:

-   Understanding the semantic intent of a query
-   Retrieving the most relevant rows via vector similarity search
-   Supplying those rows as context to an LLM

This requirement shaped nearly every design decision that followed: Omni needed to feel instant and intelligent, even on very large bases.

### Linked record recommendations: Meaning over exact matches

Semantic search also enhances a core Airtable feature: linked records. Users need relationship suggestions based on context rather than exact text matches. For instance, a project description might imply a relationship with “Team Infrastructure” without ever using that specific phrase.

Delivering these on-demand suggestions requires high-quality semantic retrieval with consistent, predictable latency.

## Our Design Priorities

To support these features and more, we anchored the system around 4 goals:

-   **Low-latency queries (500ms p99):** predictable performance is critical for user trust
-   **High-throughput writes:** bases change constantly, and embeddings must stay in sync
-   **Horizontal scalability:** the system must support millions of independent bases
-   **Self-hosting:** all customer data must remain inside Airtable-controlled infrastructure

These goals shaped every architectural decision that followed.

## Vector Database Vendor Evaluation

In late 2024, we evaluated several vector database options and ultimately selected [Milvus](https://milvus.io/) based on three key requirements.

-   First, we prioritized a self-hosted solution to ensure data privacy and maintain fine-grained control of our infrastructure.
-   Second, our write-heavy workload and bursty query patterns required a system that could scale elastically while maintaining low, predictable latency.
-   Finally, our architecture required strong isolation across millions of customer tenants.

**Milvus** emerged as the best fit: its distributed nature supports massive multi-tenancy and allows us to scale ingestion, indexing, and query execution independently, delivering performance while keeping costs predictable.

## Architecture Design

After choosing a technology, we then had to determine an architecture to represent Airtable’s unique data shape: millions of distinct “bases” owned by different customers.

## The Partitioning Challenge

We evaluated two primary data partitioning strategies:

### Option 1: Shared Partitions

Multiple bases share a partition, and queries are scoped by filtering on a base id. This improves resource utilization, but introduces additional filtering overhead and makes base deletion more complex.

### Option 2: One Base per Partition

Each Airtable base is mapped to its own physical partition in Milvus. This provides strong isolation, enables fast and simple base deletion, and avoids the performance impact of post-query filtering.

### Final Strategy

We chose option 2 for its simplicity and strong isolation. However, early tests showed that creating 100k partitions in a single Milvus collection caused significant performance degradation:

-   Partition creation latency increased from ~20 ms to ~250 ms
-   Partition load times exceeded 30 seconds

To address this, we capped the number of partitions per collection. For each Milvus cluster, we create 400 collections, each with at most 1,000 partitions. This limits the total number of bases per cluster to 400k, and new clusters are provisioned as additional customers are onboarded.

## Indexing & Recall

Index choice turned out to be one of the most consequential trade-offs in our system. When a partition is loaded, its index is cached in memory or on disk. To strike a balance between recall rate, index size, and performance, we benchmarked several index types.

-   **IVF-SQ8:** Offered a small memory footprint but lower recall.
-   **HNSW:** Delivers the best recall (99%-100%) but is memory-hungry.
-   **DiskANN:** Offers a recall similar to HNSW but with higher query latency

Ultimately, we selected HNSW for its superior recall and performance characteristics.

## The Application layer

At a high level, Airtable’s semantic search pipeline involves two core flows:

1.  **Ingestion flow:** Convert Airtable rows into embeddings and store them in Milvus
2.  **Query flow:** Embed user queries, retrieve relevant row IDs, and provide context to the LLM

Both flows must operate continuously and reliably at scale, and we walk through each below. We walk through each below.

## Ingestion Flow: Keeping Milvus in Sync with Airtable

When a user opens Omni, Airtable begins syncing their base to Milvus. We create a partition, then process the rows in chunks, generating embeddings and upserting into Milvus. From then on, we capture any changes made to the base, and re-embed and upsert those rows to keep the data consistent.

![](https://assets.zilliz.com/Airtable_Milvusblog_1_aac199ae50.png)

## Query Flow: How we use the Data

On the query side, we embed the user’s request and send it to Milvus to retrieve the most relevant row IDs. We then fetch the latest versions of those rows and include them as context in the request to the LLM.

![](https://assets.zilliz.com/Airtable_Milvusblog_2_6e9067b16d.png)

## Operational Challenges & How We Solved Them

Building a semantic search architecture is one challenge; running it reliably for hundreds of thousands of bases is another. Below are a few key operational lessons we learned along the way.

### Deployment

We deploy Milvus via its Kubernetes CRD with the [Milvus operator](https://github.com/zilliztech/milvus-operator), allowing us to define and manage clusters declaratively. Every change, whether it’s a configuration update, client improvement, or Milvus upgrade, runs through unit tests and an on-demand load test that simulates production traffic before rolling out to users.

In version 2.5, the Milvus cluster is made up of these core components:

-   Query Nodes hold the vector indices in memory and execute vector searches
-   Data Nodes handle ingestion and compaction, and persist new data to storage
-   Index Nodes build and maintain vector indexes to keep search fast as data grows
-   The Coordinator Node orchestrates all cluster activity and shard assignment
-   Proxy nodes route API traffic and balance load across nodes
-   Kafka provides the log/streaming backbone for internal messaging and data flow
-   Etcd stores cluster metadata and coordination state

With CRD-driven automation and a rigorous testing pipeline, we can roll out updates quickly and safely.

## Observability: Understanding System Health End-to-End

We monitor the system on two levels to ensure semantic search remains fast and predictable.

At the infrastructure level, we track CPU, memory usage, and pod health across all Milvus components. These signals tell us whether the cluster is operating within safe limits and help us catch issues such as resource saturation or unhealthy nodes before they affect users.

At the service layer, we focus on how well each base is keeping up with our ingestion and query workloads. Metrics like compaction and indexing throughput give us visibility into how efficiently data is being ingested. Query success rates and latency give us an understanding of the user experience querying the data, and partition growth lets us know how our data is growing, so we are alerted if we need to scale.

## Node Rotation

For security and compliance reasons, we regularly rotate Kubernetes nodes. In a vector search cluster, this is non-trivial:

-   As the query nodes are rotated, the coordinator will rebalance the in-memory data between the query nodes
-   Kafka and Etcd store stateful information and require quorum and continuous availability

We address this with strict disruption budgets and a one-node-at-a-time rotation policy. The Milvus coordinator is given time to rebalance before the next node is cycled. This careful orchestration preserves reliability without slowing down our velocity.

## Cold Partition Offloading

One of our biggest operational wins was recognizing that our data has clear hot/cold access patterns. By analyzing usage, we found that only ~25% of the data in Milvus is written to or read from in a given week. Milvus lets us offload entire partitions, freeing memory on the Query Nodes. If that data is needed later, we can reload it within seconds. This allows us to keep hot data in memory and offload the rest, reducing costs and allowing us to scale more efficiently over time.

## Data Recovery

Before rolling Milvus out broadly, we needed confidence that we could recover quickly from any failure scenario. While most issues are covered by the cluster’s built-in fault tolerance, we also planned for rare cases where data might become corrupted or the system might enter an unrecoverable state.

In those situations, our recovery path is straightforward. We first bring up a fresh Milvus cluster so we can resume serving traffic almost immediately. Once the new cluster is live, we proactively re-embed the most commonly used bases, then lazily process the rest as they are accessed. This minimizes downtime for most-accessed data while the system gradually rebuilds a consistent semantic index.

## What’s Next

Our work with [Milvus](https://milvus.io/) has laid a strong foundation for semantic search at Airtable: powering fast, meaningful AI experiences at scale. With this system in place, we’re now exploring richer retrieval pipelines and deeper AI integrations across the product. There’s a lot of exciting work ahead, and we’re just getting started.

*Thanks to all past and present Airtablets on Data Infrastructure and across the organization who contributed to this project: Alex Sorokin, Andrew Wang, Aria Malkani, Cole Dearmon-Moore, Nabeel Farooqui, Will Powelson, Xiaobing Xia.*

## About Airtable

[Airtable](https://www.airtable.com/) is a leading digital operations platform that enables organizations to build custom apps, automate workflows, and manage shared data at enterprise scale. Designed to support complex, cross-functional processes, Airtable helps teams build flexible systems for planning, coordination, and execution on a shared source of truth. As Airtable expands its AI-powered platform, technologies like Milvus play an important role in strengthening the retrieval infrastructure needed to deliver faster, smarter product experiences.

How Airtable Built and Scaled Vector Infrastructure with Milvus


Productionizing Semantic Search: How We Built and Scaled Vector Infrastructure at Airtable


Troubleshooting a Search Slowdown After Upgrading Milvus


Troubleshooting a Search Slowdown After Upgrading Milvus: Lessons from the WPS Team


Build a RAG Knowledge Base with PaddleOCR, Milvus, and ERNIE


From PDFs to Answers: Building a RAG Knowledge Base with PaddleOCR, Milvus, and ERNIE


OpenClaw Tutorial: AI Stock Agent with Exa and Milvus


I Built a Stock Monitoring Agent with OpenClaw, Exa, and Milvus for $20/Month


Will Gemini Embedding 2 kill Multi-Vector Search in Vector Databases?


Will Gemini Embedding 2 Kill Multi-Vector Search in Vector Databases?


Build a RAG Pipeline with UltraRAG v2 and Milvus


How to Build a RAG Pipeline with UltraRAG v2 and Milvus



*This blog is contributed by Jinghe Ma, a Milvus community contributor**,* *and is published here with permission.*

I wanted a chatbot that could answer questions from my own documentation, and I wanted full control over the stack behind it—from object storage to the chat interface. That led me to build a lightweight RAG chatbot with [Milvus](https://milvus.io/) and [RustFS](https://rustfs.com/) at the core.

[Milvus](https://milvus.io/) is the most widely adopted open-source vector database for building RAG applications. It separates compute from storage, keeping hot data in memory or on SSD for fast search while relying on object storage underneath for scalable, cost-efficient data management. Because it works with S3-compatible storage, it was a natural fit for this project.

For the storage layer, I chose [RustFS](https://rustfs.com/), an open-source S3-compatible object storage system written in Rust. It can be deployed via binary, Docker, or Helm chart. Although it is still in alpha and not recommended for production workloads, it was stable enough for this build.

Once the infrastructure was in place, I needed a knowledge base to query. The RustFS documentation—about 80 Markdown files—was a convenient starting point. I chunked the docs, generated embeddings, stored them in Milvus, and vibe-coded the rest: [FastAPI](https://fastapi.tiangolo.com/) for the backend and [Next.js](https://nextjs.org/) for the chat interface.

In this post, I’ll cover the full system end-to-end. The code is available at https://github.com/majinghe/chatbot. It is a working prototype rather than a production-ready system, but the goal is to provide a clear, extensible build that you can adapt for your own use. Each section below walks through one layer, from infrastructure to frontend.

## Installing Milvus and RustFS with Docker Compose

Let’s start by installing [Milvus](https://milvus.io/) and [RustFS](https://rustfs.com/).

Milvus can work with any S3-compatible object storage, although MinIO is the default backend in the standard setup. Since MinIO is no longer accepting community contributions, we’ll replace it with RustFS in this example.

To make that change, update the object storage configuration in configs/milvus.yaml inside the Milvus repository. The relevant section looks like this:

```
minio:
  address: localhost:9000
  port: 9000
  accessKeyID: rustfsadmin
  secretAccessKey: rustfsadmin
  useSSL: false
  bucketName: "rustfs-bucket"
```

There are two ways to apply this change:

-   **Mount a local config file.** Copy configs/milvus.yaml locally, update the MinIO fields to point at RustFS, then mount it into the container via a Docker volume.
-   **Patch at startup with** **yq****.** Modify the container's command to run yq against /milvus/configs/milvus.yaml before the Milvus process starts.

This build uses the first approach. The Milvus service in docker-compose.yml gets one extra volume entry:

```
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus/milvus.yaml:/milvus/configs/milvus.yaml:ro
```

### The Docker Compose Setup

The full docker-compose.yml runs four services.

**etcd** — Milvus depends on etcd for metadata storage:

```
etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.18
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3
```

**Attu** — a visual UI for Milvus, developed and open-sourced by Zilliz (note: versions after 2.6 are closed-source):

```
  attu:
    container_name: milvus-attu
    image: zilliz/attu:v2.6
    environment:
      - MILVUS_URL=milvus-standalone:19530
    ports:
      - "8000:3000"
    restart: unless-stopped
```

**RustFS** — the object storage backend:

```
rustfs:
    container_name: milvus-rustfs
    image: rustfs/rustfs:1.0.0-alpha.58
    environment:
      - RUSTFS_VOLUMES=/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_EXTERNAL_ADDRESS=:9000  # Same as internal since no port mapping
      - RUSTFS_CORS_ALLOWED_ORIGINS=*
      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
      - RUSTFS_ACCESS_KEY=rustfsadmin
      - RUSTFS_SECRET_KEY=rustfsadmin
    ports:
      - "9000:9000"# S3 API port
      - "9001:9001"# Console port
    volumes:
      - rustfs_data_0:/data/rustfs0
      - rustfs_data_1:/data/rustfs1
      - rustfs_data_2:/data/rustfs2
      - rustfs_data_3:/data/rustfs3
      - logs_data:/app/logs
    restart: unless-stopped
    healthcheck:
      test:
        [
          "CMD",
          "sh", "-c",
          "curl -f http://localhost:9000/health && curl -f http://localhost:9001/health"
        ]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
```

**Milvus** — running in standalone mode:

```
  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.6.0
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: rustfs:9000
      MQ_TYPE: woodpecker
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus/milvus.yaml:/milvus/configs/milvus.yaml:ro
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "rustfs"
```

### Starting Everything

Once the config is in place, bring up all four services:

```
docker compose -f docker-compose.yml up -d
```

You can verify everything is running with:

```
docker ps
CONTAINER ID   IMAGE                                             COMMAND                  CREATED          STATUS                      PORTS                                                                                      NAMES
4404b5cc6f7e   milvusdb/milvus:v2.6.0                            "/tini -- milvus run…"   53 minutes ago   Up 53 minutes (healthy)     0.0.0.0:9091->9091/tcp, :::9091->9091/tcp, 0.0.0.0:19530->19530/tcp, :::19530->19530/tcp   milvus-standalone
40ddc8ed08bb   zilliz/attu:v2.6                                  "docker-entrypoint.s…"   53 minutes ago   Up 53 minutes               0.0.0.0:8000->3000/tcp, :::8000->3000/tcp                                                  milvus-attu
3d2c8d80a8ce   quay.io/coreos/etcd:v3.5.18                       "etcd -advertise-cli…"   53 minutes ago   Up 53 minutes (healthy)     2379-2380/tcp                                                                              milvus-etcd
d760f6690ea7   rustfs/rustfs:1.0.0-alpha.58                      "/entrypoint.sh rust…"   53 minutes ago   Up 53 minutes (unhealthy)   0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp                              milvus-rustfs
```

With all four containers up, your services are available at:

-   **Milvus:** <ip>:19530
-   **RustFS:** <ip>:9000
-   **Attu:** <ip>:8000

## Vectorizing the RustFS Docs and Storing Embeddings in Milvus

With Milvus and RustFS running, the next step is to build the knowledge base. The source material is the RustFS Chinese documentation: 80 Markdown files that you'll chunk, embed, and store in Milvus.

### Reading and Chunking the Docs

The script recursively reads every .md file in the docs folder, then splits each file's content into chunks by newline:

```
# 3. Read Markdown files
def load_markdown_files(folder):
    files = glob.glob(os.path.join(folder, "**", "*.md"), recursive=True)
    docs = []
    for f in files:
        with open(f, "r", encoding="utf-8") as fp:
            docs.append(fp.read())
    return docs

# 4. Split documents (simple paragraph-based splitting)
def split_into_chunks(text, max_len=500):
    chunks, current = [], []
    for line in text.split("\n"):
        if len(" ".join(current)) + len(line) < max_len:
            current.append(line)
        else:
            chunks.append(" ".join(current))
            current = [line]
    if current:
        chunks.append(" ".join(current))
    return chunks
```

This chunking strategy is intentionally simple. If you want tighter control — splitting on headers, preserving code blocks, or overlapping chunks for better retrieval — this is the place to iterate.

### Embedding the Chunks

With the chunks ready, you embed them using OpenAI's text-embedding-3-large model, which outputs 3072-dimensional vectors:

```
def embed_texts(texts):
    response = client.embeddings.create(
        model="text-embedding-3-large",
        input=texts
    )
    return [d.embedding for d in response.data]
```

### Storing Embeddings in Milvus

Milvus organizes data into collections, each defined by a schema. Here, each record stores the raw text chunk alongside its embedding vector:

```
# Connect to Milvus
connections.connect("default", host="ip", port="19530")

# Define the schema
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="content", dtype=DataType.VARCHAR, max_length=2000),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=3072),
]
schema = CollectionSchema(fields, description="Markdown docs collection")

# Create the collection
if utility.has_collection("docs_collection"):
    utility.drop_collection("docs_collection")

collection = Collection(name="docs_collection", schema=schema)

# Insert data
collection.insert([all_chunks, embeddings])
collection.flush()
```

Once the insert completes, you can verify the collection in Attu at <ip>:8000 — you should see docs_collection listed under Collections.

![](https://assets.zilliz.com/blog_Milvus_Rust_FS_Vibe_Coding_Chatbot_2_a787e96076.png)

You can also check RustFS at <ip>:9000 to confirm the underlying data landed in object storage.

![](https://assets.zilliz.com/blog_Milvus_Rust_FS_Vibe_Coding_Chatbot_6_0e6d8c9471.png)

## Building a RAG Pipeline with Milvus and OpenAI’s GPT\-5

With embeddings stored in Milvus, you have everything you need to build the RAG pipeline. The flow is: embed the user's query, retrieve the most semantically similar chunks from Milvus, assemble a prompt, and call the GPT-5.The build here uses OpenAI's GPT-5, but any chat-capable model works here — the retrieval layer is what matters, and Milvus handles that regardless of which LLM generates the final answer.

```
# 1. Embed the query
query_embedding = embed_texts(query)

# 2. Retrieve similar documents from Milvus
    search_params = {"metric_type": "COSINE", "params": {"nprobe": 10}}
    results = collection.search(
        data=[query_embedding],
        anns_field="embedding",
        param=search_params,
        limit=3,
        output_fields=["content"]
    )

docs = [hit.entity.get("text") for hit in results[0]]

# 3. Assemble the RAG prompt
prompt = f"You are a RustFS expert. Answer the question based on the following documents:\n\n{docs}\n\nUser question: {query}"

# 4. Call the LLM
    response = client.chat.completions.create(
        model="gpt-5", # swap to any OpenAI model, or replace this call with another LLM provider
        messages=[{"role": "user", "content": prompt}],
        # max_tokens=16384,
        # temperature=1.0,
        # top_p=1.0,
    )

    answer = response.choices[0].message.content

    return {"answer": answer, "sources": docs}
```

To test it, run a query:

```
How do I install RustFS in Docker?
```

Query results:![](https://assets.zilliz.com/blog_Milvus_Rust_FS_Vibe_Coding_Chatbot_5_2cd609c90c.png)

![](https://assets.zilliz.com/blog_Milvus_Rust_FS_Vibe_Coding_Chatbot_3_18f4476b7a.png)

Wrapping Everything in a FastAPI + Next.js Chatbot

The RAG pipeline is working, but running a Python script every time you want to ask a question defeats the purpose. So I asked AI for a stack suggestion. The answer: **FastAPI** for the backend — the RAG code is already Python, so wrapping it in a FastAPI endpoint is the natural fit — and **Next.js** for the frontend. FastAPI exposes the RAG logic as an HTTP endpoint; Next.js calls it and renders the response in a chat window.

### FastAPI Backend

FastAPI wraps the RAG logic in a single POST endpoint. Any client can now query your knowledge base with a JSON request:

```
app = FastAPI()

@app.post("/chat")
def chat(req: ChatRequest):
    query = req.query

......
```

Start the server with:

```
uvicorn main:app --reload --host 0.0.0.0 --port 9999
INFO:     Will watch for changes in these directories: ['/home/xiaomage/milvus/chatbot/.venv']
INFO:     Uvicorn running on http://0.0.0.0:9999 (Press CTRL+C to quit)
INFO:     Started reloader process [2071374] using WatchFiles
INFO:     Started server process [2071376]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
```

### Next.js Frontend

The frontend sends the user's query to the FastAPI endpoint and renders the response. The core fetch logic:

javascript

```
   try {
      const res = await fetch('http://localhost:9999/chat', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ query: input }),
      });

      const data = await res.json();
      const botMessage: Message = { role: 'bot', content: data.answer || 'No response' };
      setMessages(prev => [...prev, userMessage, botMessage]);
    } catch (error) {
      console.error(error);
      const botMessage: Message = { role: 'bot', content: 'Error connecting to server.' };
      setMessages(prev => [...prev, botMessage]);
    } finally {
      setLoading(false);
    }
```

Start the frontend with:

```
pnpm run dev -H 0.0.0.0 -p 3000

> rag-chatbot@0.1.0 dev /home/xiaomage/milvus/chatbot-web/rag-chatbot
> next dev --turbopack -H 0.0.0.0 -p 3000

   ▲ Next.js 15.5.3 (Turbopack)
   - Local:        http://localhost:3000
   - Network:      http://0.0.0.0:3000

 ✓ Starting...
 ✓ Ready in 1288ms
```

Open `http://<ip>:3000/chat` in your browser.

![](https://assets.zilliz.com/blog_Milvus_Rust_FS_Vibe_Coding_Chatbot_1_0832811fc8.png)Type a question:

```
How do I install RustFS in Docker?
```

Chat interface response::

![](https://assets.zilliz.com/blog_Milvus_Rust_FS_Vibe_Coding_Chatbot_4_91d679ae50.png)

And that’s the chatbot done.

## Conclusion

What started as curiosity about Milvus's storage backend turned into a full working RAG chatbot — and the path from one to the other was shorter than expected. Here's what the build covers, end to end:

-   **[Milvus](http://milvus.io)** **+** **[RustFS](https://rustfs.com/)** **with Docker Compose.** Milvus runs in standalone mode with RustFS as its object storage backend, replacing the default MinIO. Four services total: etcd, Milvus, RustFS, and Attu.
-   **Vectorizing the knowledge base.** The RustFS documentation — 80 Markdown files — gets chunked, embedded with text-embedding-3-large, and stored in Milvus as 466 vectors.
-   **The RAG pipeline.** At query time, the user's question gets embedded the same way, Milvus retrieves the three most semantically similar chunks, and GPT-5 generates an answer grounded in those docs.
-   **The chatbot UI.** FastAPI wraps the pipeline in a single POST endpoint; Next.js puts a chat window in front of it. No more dropping into a terminal to ask a question.

A few of my takeaways from the process:

-   **[Milvus's documentation](https://milvus.io/docs)** **is great.** Especially the deployment sections — clear, complete, easy to follow.
-   **[RustFS](https://rustfs.com/)** **is a pleasure to work with as an object storage backend.** Dropping it in for MinIO took less effort than expected.
-   **Vibe coding is fast, right up until scope takes over.** One thing kept leading to another — Milvus to RAG to chatbot to "maybe I should Dockerize the whole thing." Requirements don't converge on their own.
-   **Debugging teaches more than reading.** Every failure in this build made the next section click faster than any documentation would have.

All the code from this build is at [](https://github.com/majinghe/chatbot)[github.com/majinghe/chatbot](https://github.com/majinghe/chatbot). If you want to try [Milvus](http://milvus.io) yourself, the [quickstart](https://milvus.io/docs/quickstart.md) is a good place to start. If you want to talk through what you're building or run into something unexpected, come and find us in the [Milvus Slack](https://milvus.io/slack). If you'd rather have a dedicated conversation, you can also [book a slot at office hours](https://meetings.hubspot.com/chloe-williams1/milvus-office-hour?uuid=4cb203e5-482a-47e0-90a6-7acc511d61f4).


Milvus + RustFS: Build a Lightweight RAG Chatbot


Milvus + RustFS+ Vibe Coding: Build a Lightweight RAG Chatbot from Scratch



Nowadays, you can upload a PDF to any modern LLM and ask questions about it. For a handful of documents, that works fine. But most LLMs cap out at a few hundred pages of context, so a large corpus simply won't fit. Even when it does fit, you're paying to process every page on every query. Ask a hundred questions about the same 500-page document set, and you pay for 500 pages a hundred times over. That gets expensive fast.

Retrieval-augmented generation (RAG) solves this by separating indexing from answering. You encode your documents once, store the representations in a vector database, and at query time you retrieve only the most relevant pages to send to the LLM. The model reads three pages per query, not your entire corpus. That makes it practical to build document Q&A over collections that keep growing.

This tutorial walks you through building a multimodal RAG pipeline with three openly licensed components:

-   **[ColQwen2](https://huggingface.co/vidore/colqwen2-v1.0-merged)** [](https://huggingface.co/vidore/colqwen2-v1.0-merged)encodes each PDF page as an image into multi-vector embeddings, replacing the traditional OCR and text chunking step.
-   **[Milvus](http://milvus.io)** stores those vectors and handles similarity search at query time, retrieving only the most relevant pages.
-   **[Qwen3.5-397B-A17B](https://qwen.ai/blog?id=qwen3.5)** reads the retrieved page images and generates an answer based on what it sees.

By the end, you'll have a working system that takes a PDF and a question, finds the most relevant pages, and returns an answer grounded in what the model sees.

## What is Multimodal RAG?

The intro covered why RAG matters at scale. The next question is what kind of RAG you need, because the traditional approach has a blind spot.

Traditional RAG extracts text from documents, embeds it as vectors, retrieves the closest matches at query time, and passes those text chunks to an LLM. That works well for text-heavy content with clean formatting. It breaks when your documents contain:

-   Tables, where meaning depends on the relationship between rows, columns, and headers.
-   Charts and diagrams, where information is entirely visual and has no text equivalent.
-   Scanned documents or handwritten notes, where OCR output is unreliable or incomplete.

Multimodal RAG replaces text extraction with image encoding. You render each page as an image, encode it with a vision-language model, and retrieve page images at query time. The LLM sees the original page — tables, figures, formatting and all — and answers based on what it sees.

![](https://assets.zilliz.com/blog_Col_Qwen2_Milvus_Qwen3_5397_BA_17_B_5_2f55d33896.png)

## Structure of Multimodal RAG Pipeline: ColQwen2 for Encoding, Milvus for Search, Qwen3.5 for Generation

### How the Pipeline Works ![](https://assets.zilliz.com/blog_Col_Qwen2_Milvus_Qwen3_5397_BA_17_B_2_84c822b9ef.png)

### Tech Stack

| **Component** | **Choice** | **Role** |
| --- | --- | --- |
| PDF processing | pdf2image + poppler | Renders PDF pages as high-resolution images |
| Embedding model | [colqwen2-v1.0](https://huggingface.co/vidore/colqwen2-v1.0-merged) | Vision-language model; encodes each page into ~755 128-dim patch vectors |
| Vector database | [Milvus Lite](https://milvus.io/) | Stores patch vectors and handles similarity search; runs locally with no server setup |
| Generation model | [Qwen3.5-397B-A17B](https://qwen.ai/blog?id=qwen3.5) | Multimodal LLM called via OpenRouter API; reads retrieved page images to generate answers |

## Step-by-Step Implementation for Multi-Modal RAG with ColQwen2+ Milvus+ Qwen3.5-397B-A17B

### Environment Setup

1.  Install Python Dependencies

```
pip install colpali-engine pymilvus openai pdf2image torch pillow tqdm
```

2.  Install Poppler, the PDF Rendering Engine

```
# macOS
brew install poppler

# Ubuntu/Debian
sudo apt-get install poppler-utils

# Windows: download from https://github.com/oschwartz10612/poppler-windows

```

3.  Download the Embedding Model, ColQwen2

Download vidore/colqwen2-v1.0-merged from HuggingFace (~4.4 GB) and save it locally:

```
mkdir -p ~/models/colqwen2-v1.0-merged
# Download all model files to this directory
```

4.  Get an OpenRouter API Key

Sign up and generate a key at [](https://openrouter.ai/settings/keys)[https://openrouter.ai/settings/keys](https://openrouter.ai/settings/keys).

### Step 1: Import Dependencies and Configure

```
import os, io, base64
import torch
import numpy as np
from PIL import Image
from tqdm import tqdm
from pdf2image import convert_from_path

from openai import OpenAI
from pymilvus import MilvusClient, DataType
from colpali_engine.models import ColQwen2, ColQwen2Processor

# --- Configuration ---
EMBED_MODEL = os.path.expanduser("~/models/colqwen2-v1.0-merged")
EMBED_DIM = 128              # ColQwen2 output vector dimension
MILVUS_URI = "./milvus_demo.db"  # Milvus Lite local file
COLLECTION = "doc_patches"
TOP_K = 3                    # Number of pages to retrieve
CANDIDATE_PATCHES = 300      # Candidate patches per query token

# OpenRouter LLM config
OPENROUTER_API_KEY = os.environ.get(
    "OPENROUTER_API_KEY",
    "<your-api-key-here>",
)
GENERATION_MODEL = "qwen/qwen3.5-397b-a17b"

# Device selection
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
DTYPE = torch.bfloat16 if DEVICE == "cuda" else torch.float32
print(f"Device: {DEVICE}")
```

Output: Device: cpu

### Step 2: Load the Embedding Model

**ColQwen2** is a vision-language model that encodes document images into ColBERT-style multi-vector representations. Each page produces several hundred 128-dimensional patch vectors.

```
print(f"Loading embedding model: {EMBED_MODEL}")
emb_model = ColQwen2.from_pretrained(
    EMBED_MODEL,
    torch_dtype=DTYPE,
    attn_implementation="flash_attention_2" if DEVICE == "cuda" else None,
    device_map=DEVICE,
).eval()
emb_processor = ColQwen2Processor.from_pretrained(EMBED_MODEL)
print(f"Embedding model ready on {DEVICE}")
```

Output:

![](https://assets.zilliz.com/blog_Col_Qwen2_Milvus_Qwen3_5397_BA_17_B_1_1fbbeba04e.png)

### Step 3: Initialize Milvus

This tutorial uses Milvus Lite, which runs as a local file with zero configuration — no separate server process needed.

**Database schema:**

**id**: INT64, auto-increment primary key

**doc_id**: INT64, page number (which page of the PDF)

**patch_idx**: INT64, patch index within that page

**vector**: FLOAT_VECTOR(128), the patch's 128-dimensional embedding

```
milvus_client = MilvusClient(uri=MILVUS_URI)

if milvus_client.has_collection(COLLECTION):
    milvus_client.drop_collection(COLLECTION)

schema = milvus_client.create_schema(auto_id=True, enable_dynamic_field=True)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("doc_id", DataType.INT64)
schema.add_field("patch_idx", DataType.INT64)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=EMBED_DIM)

index = milvus_client.prepare_index_params()
index.add_index(field_name="vector", index_type="FLAT", metric_type="IP")
milvus_client.create_collection(COLLECTION, schema=schema, index_params=index)
print("Milvus collection created.")
```

Output: Milvus collection created.

### Step 4: Convert PDF Pages to Images

You render each page at 150 DPI. No text extraction happens here — the pipeline treats every page purely as an image.

```
PDF_PATH = "Milvus vs Zilliz.pdf"  # Replace with your own PDF
images = [p.convert("RGB") for p in convert_from_path(PDF_PATH, dpi=150)]
print(f"{len(images)} pages loaded.")

# Preview the first page
images[0].resize((400, int(400 * images[0].height / images[0].width)))
```

Output:

![](https://assets.zilliz.com/blog_Col_Qwen2_Milvus_Qwen3_5397_BA_17_B_4_8720da8494.png)

### Step 5: Encode Images and Insert into Milvus

ColQwen2 encodes each page into multi-vector patch embeddings. You then insert every patch as a separate row in Milvus.

```
# Encode all pages
all_page_embs = []
with torch.no_grad():
    for i in tqdm(range(0, len(images), 2), desc="Encoding pages"):
        batch = images[i : i + 2]
        inputs = emb_processor.process_images(batch).to(emb_model.device)
        embs = emb_model(**inputs)
        for e in embs:
            all_page_embs.append(e.cpu().float().numpy())

print(f"Encoded {len(all_page_embs)} pages, ~{all_page_embs[0].shape[0]} patches per page, dim={all_page_embs[0].shape[1]}")
```

Output: Encoded 17 pages, ~755 patches per page, dim=128

```
# Insert into Milvus
for doc_id, patch_vecs in enumerate(all_page_embs):
    rows = [
        {"doc_id": doc_id, "patch_idx": j, "vector": v.tolist()}
        for j, v in enumerate(patch_vecs)
    ]
    milvus_client.insert(COLLECTION, rows)

total = milvus_client.get_collection_stats(COLLECTION)["row_count"]
print(f"Indexed {len(all_page_embs)} pages, {total} patches total.")
```

Output: Indexed 17 pages, 12835 patches total.

A 17-page PDF produces 12,835 patch vector records — roughly 755 patches per page.

### Step 6: Retrieve — Query Encoding + MaxSim Reranking

This is the core retrieval logic. It works in three stages:

**Encode the query** into multiple token vectors.

**Search Milvus** for each token vector's closest patches.

**Aggregate by page** using MaxSim: for each query token, take the highest-scoring patch in each page, then sum those scores across all tokens. The page with the highest total score is the best match.

**How MaxSim works:** For each query token vector, you find the document patch with the highest inner product (the "max" in MaxSim). You then sum these maximum scores across all query tokens to get a total relevance score per page. Higher score = stronger semantic match between the query and the page's visual content.

```
question = "What is the difference between Milvus and Zilliz Cloud?"

# 1. Encode the query
with torch.no_grad():
    query_inputs = emb_processor.process_queries([question]).to(emb_model.device)
    query_vecs = emb_model(**query_inputs)[0].cpu().float().numpy()
print(f"Query encoded: {query_vecs.shape[0]} token vectors")

# 2. Search Milvus for each query token vector
doc_patch_scores = {}
for qv in query_vecs:
    hits = milvus_client.search(
        COLLECTION, data=[qv.tolist()], limit=CANDIDATE_PATCHES,
        output_fields=["doc_id", "patch_idx"],
        search_params={"metric_type": "IP"},
    )[0]
    for h in hits:
        did = h["entity"]["doc_id"]
        pid = h["entity"]["patch_idx"]
        score = h["distance"]
        doc_patch_scores.setdefault(did, {})[pid] = max(
            doc_patch_scores.get(did, {}).get(pid, 0), score
        )

# 3. MaxSim aggregation: total score per page = sum of all matched patch scores
doc_scores = {d: sum(ps.values()) for d, ps in doc_patch_scores.items()}
ranked = sorted(doc_scores.items(), key=lambda x: x[1], reverse=True)[:TOP_K]
print(f"Top-{TOP_K} retrieved pages: {[(d, round(s, 2)) for d, s in ranked]}")

```

Output:

```
Query encoded: 24 token vectors
Top-3 retrieved pages: [(16, 161.16), (12, 135.73), (7, 122.58)]
```

```
# Display the retrieved pages
context_images = [images[d] for d, _ in ranked if d < len(images)]
for i, img in enumerate(context_images):
    print(f"--- Retrieved page {ranked[i][0]} (score: {ranked[i][1]:.2f}) ---")
    display(img.resize((500, int(500 * img.height / img.width))))
```

![](https://assets.zilliz.com/blog_Col_Qwen2_Milvus_Qwen3_5397_BA_17_B_6_2842a54af8.png)

### Step 7: Generate an Answer with the Multimodal LLM

You send the retrieved page images — not extracted text — along with the user's question to Qwen3.5. The LLM reads the images directly to produce an answer.

```
def image_to_uri(img):
    """Convert an image to a base64 data URI for sending to the LLM."""
    img = img.copy()
    w, h = img.size
    if max(w, h) > 1600:
        r = 1600 / max(w, h)
        img = img.resize((int(w * r), int(h * r)), Image.LANCZOS)
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    return f"data:image/png;base64,{base64.b64encode(buf.getvalue()).decode()}"

# Build the multimodal prompt
context_images = [images[d] for d, _ in ranked if d < len(images)]
content = [
    {"type": "image_url", "image_url": {"url": image_to_uri(img)}}
    for img in context_images
]
content.append({
    "type": "text",
    "text": (
        f"Above are {len(context_images)} retrieved document pages.\n"
        f"Read them carefully and answer the following question:\n\n"
        f"Question: {question}\n\n"
        f"Be concise and accurate. If the documents don't contain "
        f"relevant information, say so."
    ),
})

# Call the LLM
llm = OpenAI(api_key=OPENROUTER_API_KEY, base_url="https://openrouter.ai/api/v1")
response = llm.chat.completions.create(
    model=GENERATION_MODEL,
    messages=[{"role": "user", "content": content}],
    max_tokens=1024,
    temperature=0.7,
)
answer = response.choices[0].message.content.strip()
print(f"Question: {question}\n")
print(f"Answer: {answer}")
```

Results:  
![](https://assets.zilliz.com/blog_Col_Qwen2_Milvus_Qwen3_5397_BA_17_B_3_33fa5d551d.png)

## Conclusion

In this tutorial, we built a multimodal RAG pipeline that takes a PDF, converts each page into an image, encodes those images into multi-vector patch embeddings with ColQwen2, stores them in Milvus, and retrieves the most relevant pages at query time using MaxSim scoring. Instead of extracting text and hoping OCR preserves the layout, the pipeline sends the original page images to Qwen3.5, which reads them visually and generates an answer.

This tutorial is a starting point, not a production deployment. A few things to keep in mind as you take it further.

On tradeoffs:

-   **Storage scales with page count.** Each page produces ~755 vectors, so a 1,000-page corpus means roughly 755,000 rows in Milvus. The FLAT index used here works for demos but you'd want IVF or HNSW for larger collections.
-   **Encoding is slower than text embedding.** ColQwen2 is a 4.4 GB vision model. Encoding images takes longer per page than embedding text chunks. For a batch indexing job that runs once, this is usually fine. For real-time ingestion it's worth benchmarking.
-   **This approach works best for visually rich documents.** If your PDFs are mostly clean, single-column text with no tables or figures, traditional text-based RAG may retrieve more precisely and cost less to run.

On what to try next:

-   **Swap in a different multimodal LLM.** This tutorial uses Qwen3.5 via OpenRouter, but the retrieval pipeline is model-agnostic. You could point the generation step at GPT-4o, Gemini, or any multimodal model that accepts image inputs.
-   **Scale up [Milvus](http://milvus.io).** Milvus Lite runs as a local file, which is great for prototyping. For production workloads, Milvus on Docker/Kubernetes or Zilliz Cloud (fully managed Milvus) handles larger corpora without you managing infrastructure.
-   **Experiment with different document types.** The pipeline here uses a comparison PDF, but it works the same way on scanned contracts, engineering drawings, financial statements, or research papers with dense figures.

To get started, install [Milvus Lite](https://github.com/milvus-io/milvus-lite) with pip install pymilvus and grab the ColQwen2 weights from HuggingFace.

Got questions, or want to show off what you've built? The [Milvus Slack](https://milvus.io/slack) is the fastest way to get help from the community and the team. If you'd prefer a one-on-one conversation, you can book time at our [office hours](https://meetings.hubspot.com/chloe-williams1/milvus-office-hour?uuid=4cb203e5-482a-47e0-90a6-7acc511d61f4).

## Keep Reading

-    [What If You Could See Why RAG Fails? Debugging RAG in 3D with Project_Golem and Milvus](https://milvus.io/blog/debugging-rag-in-3d-with-projectgolem-and-milvus.md)
    
-    [Is RAG Becoming Outdated Now That Long-Running Agents Like Claude Cowork Are Emerging?](https://milvus.io/blog/is-rag-become-outdated-now-long-running-agents-like-claude-cowork-are-emerging.md)
    
-    [How We Built a Semantic Highlighting Model for RAG Context Pruning and Token Saving](https://milvus.io/blog/semantic-highlighting-model-for-rag-context-pruning-and-token-saving.md)
    
-    [AI Code Review Gets Better When Models Debate: Claude vs Gemini vs Codex vs Qwen vs MiniMax](https://milvus.io/blog/ai-code-review-gets-better-when-models-debate-claude-vs-gemini-vs-codex-vs-qwen-vs-minimax.md)


How to Build Multimodal RAG with ColQwen2, Milvus, Qwen3.5


How to Build Multimodal RAG with ColQwen2, Milvus, and Qwen3.5


Nano Banana 2 + Milvus: E-Commerce AI Image Generation Tutorial


Build a Bestseller-to-Image Pipeline for E-Commerce with Nano Banana 2 + Milvus + Qwen 3.5


More and more teams are building AI agents, and the tasks they assign to them are becoming more complex. Many real-world workflows involve long-running jobs with multiple steps and many tool calls. As these tasks grow, two problems appear quickly: higher token costs and the limits of the model’s context window. Agents also often need to remember information across sessions, such as past research results, user preferences, or earlier conversations.

Frameworks like [**Deep Agents**](https://docs.langchain.com/oss/python/deepagents/overview), released by LangChain, help organize these workflows. It provides a structured way to run agents, with support for task planning, file access, and sub-agent delegation. This makes it easier to build agents that can handle long, multi-step tasks more reliably.

But workflows alone are not enough. Agents also need **long-term memory** so they can retrieve useful information from previous sessions. This is where [**Milvus**](https://milvus.io/), an open-source vector database, comes in. By storing embeddings of conversations, documents, and tool results, Milvus allows agents to search and recall past knowledge.

In this article, we’ll explain how Deep Agents works and show how to combine it with Milvus to build AI agents with structured workflows and long-term memory.

## What Is Deep Agents?

**Deep Agents** is an open-source agent framework built by the LangChain team. It is designed to help agents handle long-running, multi-step tasks more reliably. It focuses on three main capabilities:

**1. Task Planning**

Deep Agents includes built-in tools like `write_todos` and `read_todos`. The agent breaks a complex task into a clear to-do list, then works through each item step by step, marking tasks as completed. 

**2. File System Access**

It provides tools such as `ls`, `read_file`, and `write_file`, so the agent can view, read, and write files. If a tool produces a large output, the result is automatically saved to a file instead of staying in the model’s context window. This helps prevent the context window from filling up.

**3. Sub-agent Delegation**

Using a `task` tool, the main agent can hand off subtasks to specialized sub-agents. Each sub-agent has its own context window and tools, which helps keep work organized.

![](https://assets.zilliz.com/1_59401bc198.png)

Technically, an agent created with `create_deep_agent` is a compiled **LangGraph StateGraph**. (LangGraph is the workflow library developed by the LangChain team, and StateGraph is its core state structure.) Because of this, Deep Agents can directly use LangGraph features like streaming output, checkpointing, and human-in-the-loop interaction.

**So what makes Deep Agents useful in practice?**

Long-running agent tasks often face problems such as context limits, high token costs, and unreliable execution. Deep Agents helps solve these issues by making agent workflows more structured and easier to manage. By reducing unnecessary context growth, it lowers token usage and keeps long-running tasks more cost-efficient.

It also makes complex, multi-step tasks easier to organize. Subtasks can run independently without interfering with each other, which improves reliability. At the same time, the system is flexible, allowing developers to customize and extend it as their agents grow from simple experiments to production applications.

## Customization in Deep Agents

A general framework cannot cover every industry or business need. Deep Agents is designed to be flexible, so developers can adjust it to fit their own use cases.

With customization, you can:

-   Connect your own internal tools and APIs
    
-   Define domain-specific workflows
    
-   Make sure the agent follows business rules
    
-   Support memory and knowledge sharing across sessions
    

Here are the main ways you can customize Deep Agents:

### System Prompt Customization

You can add your own system prompt on top of the default instructions provided by middleware. This is useful for defining domain rules and workflows.

A good custom prompt may include:

-   **Domain workflow rules**
    
Example: “For data analysis tasks, always run exploratory analysis before building a model.”

-   **Specific examples**
    
Example: “Combine similar literature search requests into one todo item.”

-   **Stopping rules**
    
Example: “Stop if more than 100 tool calls are used.”

-   **Tool coordination guidance**
    
Example: “Use `grep` to find code locations, then use `read_file` to view details.”

Avoid repeating instructions that middleware already handles, and avoid adding rules that conflict with the default behavior.

### Tools

You can add your own tools to the built-in toolset. Tools are defined as normal Python functions, and their docstrings describe what they do.

```
from deepagents import create_deep_agent
def internet_search(query: str) -> str:
    """Run a web search"""
    return tavily_client.search(query)
agent = create_deep_agent(tools=[internet_search])
```

Deep Agents also supports tools that follow the Model Context Protocol (MCP) standard through `langchain-mcp-adapters`.

```
from langchain_mcp_adapters.client import MultiServerMCPClient
from deepagents import create_deep_agent
async def main():
    mcp_client = MultiServerMCPClient(...)
    mcp_tools = await mcp_client.get_tools()
    agent = create_deep_agent(tools=mcp_tools)
    async for chunk in agent.astream({"messages": [{"role": "user", "content": "..."}]}):
        chunk["messages"][-1].pretty_print()
```

### Middleware

You can write custom middleware to:

-   Add or modify tools
    
-   Adjust prompts
    
-   Hook into different stages of the agent’s execution
    
```
from langchain_core.tools import tool
from deepagents import create_deep_agent
from deepagents.middleware import AgentMiddleware
@tool
def get_weather(city: str) -> str:
    """Get the weather in a city."""
    return f"The weather in {city} is sunny."
class WeatherMiddleware(AgentMiddleware):
    tools = [get_weather]
agent = create_deep_agent(middleware=[WeatherMiddleware()])
```

Deep Agents also includes built-in middleware for planning, sub-agent management, and execution control.

| Middleware | Function |
| --- | --- |
| TodoListMiddleware | Provides write_todos and read_todos tools to manage task lists |
| FilesystemMiddleware | Provides file operation tools and automatically saves large tool outputs |
| SubAgentMiddleware | Provides the task tool to delegate work to sub-agents |
| SummarizationMiddleware | Automatically summarizes when context exceeds 170k tokens |
| AnthropicPromptCachingMiddleware | Enables prompt caching for Anthropic models |
| PatchToolCallsMiddleware | Fixes incomplete tool calls caused by interruptions |
| HumanInTheLoopMiddleware | Configures tools that require human approval |

### Sub-agents

The main agent can delegate subtasks to sub-agents using the `task` tool. Each sub-agent runs in its own context window and has its own tools and system prompt.

```
from deepagents import create_deep_agent
research_subagent = {
    "name": "research-agent",
    "description": "Used to research in-depth questions",
    "prompt": "You are an expert researcher",
    "tools": [internet_search],
    "model": "openai:gpt-4o",  # Optional, defaults to main agent model
}
agent = create_deep_agent(subagents=[research_subagent])
```

For advanced use cases, you can even pass in a pre-built LangGraph workflow as a sub-agent.

```
from deepagents import CompiledSubAgent, create_deep_agent
custom_graph = create_agent(model=..., tools=..., prompt=...)
agent = create_deep_agent(
    subagents=[CompiledSubAgent(
        name="data-analyzer",
        description="Specialized agent for data analysis",
        runnable=custom_graph
    )]
)
```

### `interrupt_on` (Human Approval Control)

You can specify certain tools that require human approval using the `interrupt_on` parameter. When the agent calls one of these tools, execution pauses until a person reviews and approves it.

```
from langchain_core.tools import tool
from deepagents import create_deep_agent
from langgraph.checkpoint.memory import MemorySaver
@tool
def delete_file(path: str) -> str:
    """Delete a file from the filesystem."""
    return f"Deleted {path}"
agent = create_deep_agent(
    tools=[delete_file],
    interrupt_on={
        "delete_file": {
            "allowed_decisions": ["approve", "edit", "reject"]
        }
    },
    checkpointer=MemorySaver()
)
```

### Backend Customization (Storage)

You can choose different storage backends to control how files are handled. Current options include:

-   **StateBackend** (temporary storage)
    
-   **FilesystemBackend** (local disk storage)
    
```
StoreBackend(persistent storage)、CompositeBackend(hybrid routing)。
from deepagents import create_deep_agent
from deepagents.backends import FilesystemBackend
agent = create_deep_agent(
    backend=FilesystemBackend(root_dir="/path/to/project")
)
```

By changing the backend, you can adjust file storage behavior without changing the overall system design.

## Why Use Deep Agents with Milvus for AI Agents?

In real applications, agents often need memory that lasts across sessions. For example, they may need to remember user preferences, build up domain knowledge over time, record feedback to adjust behavior, or keep track of long-term research tasks.

By default, Deep Agents uses `StateBackend`, which only stores data during a single session. When the session ends, everything is cleared. This means it cannot support long-term, cross-session memory.

To enable persistent memory, we use [**Milvus**](https://milvus.io/) as the vector database together with `StoreBackend`. Here’s how it works: important conversation content and tool results are converted into embeddings (numerical vectors that represent meaning) and stored in Milvus. When a new task starts, the agent performs semantic search to retrieve related past memories. This allows the agent to “remember” relevant information from previous sessions.

Milvus is well suited for this use case because of its compute-storage separation architecture. It supports:

-   Horizontal scaling to tens of billions of vectors
    
-   High-concurrency queries
    
-   Real-time data updates
    
-   Production-ready deployment for large-scale systems
    
Technically, Deep Agents uses `CompositeBackend` to route different paths to different storage backends:

| Path | Backend | Purpose |
| --- | --- | --- |
| /workspace/, /temp/ | StateBackend | Temporary data, cleared after the session |
| /memories/, /knowledge/ | StoreBackend + Milvus | Persistent data, searchable across sessions |

With this setup, developers only need to save long-term data under paths like `/memories/`. The system automatically handles cross-session memory. Detailed configuration steps are provided in the section below.

## Hands-on: Build an AI Agent with Long-Term Memory Using Milvus and Deep Agents

This example shows how to give a DeepAgents-based agent persistent memory using Milvus.

### Step 1: Install dependencies

```
pip install deepagents tavily-python langchain-milvus
```

### Step 2: Set up the memory backend

```
from deepagents.backends import CompositeBackend, StateBackend, StoreBackend
from langchain_milvus.storage import MilvusStore
# from langgraph.store.memory import InMemoryStore # for testing only
# Configure Milvus storage
milvus_store = MilvusStore(
    collection_name="agent_memories",
    embedding_service=... # embedding is required here, or use MilvusStore default configuration
)
backend = CompositeBackend(
    default=StateBackend(),
    routes={"/memories/": StoreBackend(store=InMemoryStore())} 
)
```

### Step 3: Create the agent

```
from tavily import TavilyClient
import os
tavily_client = TavilyClient(api_key=os.environ["TAVILY_API_KEY"])
def internet_search(query: str, max_results: int = 5) -> str:
    """Perform an internet search"""
    results = tavily_client.search(query, max_results=max_results)
    return "\n".join([f"{r['title']}: {r['content']}" for r in results["results"]])
agent = create_deep_agent(
    tools=[internet_search],
    system_prompt="You are a research expert. Write important findings to the /memories/ directory for cross-session reuse.",
    backend=backend
)
# Run the agent
result = agent.invoke({
    "messages": [{"role": "user", "content": "Research the technical features of the Milvus vector database"}]
})
```

**Key Points**

-   **Persistent path**
    
Any files saved under `/memories/` will be stored permanently and can be accessed across different sessions.

-   **Production setup**
    
The example uses `InMemoryStore()` for testing. In production, replace it with a Milvus adapter to enable scalable semantic search.

-   **Automatic memory**
    
The agent automatically saves research results and important outputs to the `/memories/` folder. In later tasks, it can search and retrieve relevant past information.

## Built-in Tools Overview

Deep Agents includes several built-in tools, provided through middleware. They fall into three main groups:

### Task Management (`TodoListMiddleware`)

-   `write_todos`
    
Creates a structured todo list. Each task can include a description, priority, and dependencies.

-   `read_todos`
    
Shows the current todo list, including completed and pending tasks.

### File System Tools (`FilesystemMiddleware`)

-   `ls`
    
Lists files in a directory. Must use an absolute path (starting with `/`).

-   `read_file`
    
Reads file content. Supports `offset` and `limit` for large files.

-   `write_file`
    
Creates or overwrites a file.

-   `edit_file`
    
Replaces specific text inside a file.

-   `glob`
    
Finds files using patterns, such as `**/*.py` to search for all Python files.

-   `grep`
    
Searches for text inside files.

-   `execute`
    
Runs shell commands in a sandbox environment. Requires the backend to support `SandboxBackendProtocol`.

### Sub-agent Delegation (`SubAgentMiddleware`)

-   `task`
    
Sends a subtask to a specific sub-agent. You provide the sub-agent name and the task description.

### How Tool Outputs Are Handled

If a tool generates a large result, Deep Agents automatically saves it to a file.

For example, if `internet_search` returns 100KB of content, the system saves it to something like `/tool_results/internet_search_1.txt`. The agent keeps only the file path in its context. This reduces Token usage and keeps the context window small.

## DeepAgents vs. Agent Builder: When Should You Use Each?

*Since this article focuses on DeepAgents, it's also helpful to understand how it compares with* [*Agent Builder*](https://www.langchain.com/langsmith/agent-builder)*, another agent-building option in the LangChain ecosystem.*

LangChain offers several ways to build AI agents, and the best choice usually depends on how much control you want over the system.

**DeepAgents** is designed for building autonomous agents that handle long-running, multi-step tasks. It gives developers full control over how the agent plans tasks, uses tools, and manages memory. Because it is built on LangGraph, you can customize components, integrate Python tools, and modify the storage backend. This makes DeepAgents a good fit for complex workflows and production systems where reliability and flexibility are important.

**Agent Builder**, in contrast, focuses on ease of use. It hides most of the technical details, so you can describe an agent, add tools, and run it quickly. Memory, tool usage, and human approval steps are handled automatically. This makes Agent Builder useful for quick prototypes, internal tools, or early experiments.

**Agent Builder and DeepAgents are not separate systems—they are part of the same stack.** Agent Builder is built on top of DeepAgents. Many teams start with Agent Builder to test ideas, then switch to DeepAgents when they need more control. Workflows created with DeepAgents can also be turned into Agent Builder templates so others can reuse them easily.

## Conclusion

Deep Agents makes complex agent workflows easier to manage by using three main ideas: task planning, file storage, and sub-agent delegation. These mechanisms turn messy, multi-step processes into structured workflows. When combined with Milvus for vector search, the agent can also keep long-term memory across sessions.

For developers, this means lower Token costs and a more reliable system that can scale from a simple demo to a production environment.

If you're building AI agents that need structured workflows and real long-term memory, we’d love to connect.

Have questions about Deep Agents or using Milvus as a persistent memory backend? Join our [Slack channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session to discuss your use case.


How to Build Production-Ready AI Agents with Deep Agents


How to Build Production-Ready AI Agents with Deep Agents and Milvus



If you’ve spent any time with [OpenClaw](https://milvus.io/blog/openclaw-formerly-clawdbot-moltbot-explained-a-complete-guide-to-the-autonomous-ai-agent.md) (formerly Clawdbot and Moltbot), you already know how good this AI Agent is. It’s fast, local, flexible, and capable of pulling off surprisingly complex workflows across Slack, Discord, your codebase, and practically anything else you hook it into. But once you start using it seriously, one pattern quickly emerges: **your token usage starts to climb.**

This isn’t OpenClaw’s fault specifically — it’s how most AI agents behave today. They trigger an LLM call for almost everything: looking up a file, planning a task, writing a note, executing a tool, or asking a follow-up question. And because tokens are the universal currency of these calls, every action has a cost.

To understand where that cost comes from, we need to look under the hood at two big contributors:

-   **Search:** Badly constructed searches pull in sprawling context payloads — entire files, logs, messages, and code regions that the model didn’t actually need.
-   **Memory:** Storing unimportant information forces the agent to reread and reprocess it on future calls, compounding token usage over time.

Both issues silently increase operational costs without improving capability.

## How AI Agents Like OpenClaw Actually Perform Searches — and Why That Burns Tokens

When an agent needs information from your codebase or document library, it typically does the equivalent of a project-wide **Ctrl+F**. Every matching line is returned — unranked, unfiltered, and unprioritized. Claude Code implements this through a dedicated Grep tool built on ripgrep. OpenClaw doesn't have a built-in codebase search tool, but its exec tool lets the underlying model run any command, and loaded skills can guide the agent to use tools like rg. In both cases, codebase search returns keyword matches unranked and unfiltered.

This brute-force approach works fine in small projects. But as repositories grow, so does the price. Irrelevant matches pile into the LLM’s context window, forcing the model to read and process thousands of tokens it didn’t actually need. A single unscoped search might drag in full files, huge comment blocks, or logs that share a keyword but not the underlying intent. Repeat that pattern across a long debugging or research session, and the bloat adds up quickly.

Both OpenClaw and Claude Code try to manage this growth. OpenClaw prunes oversized tool outputs and compacts long conversation histories, while Claude Code limits file-read output and supports context compaction. These mitigations work — but only after the bloated query has already been executed. The unranked search results still consumed tokens, and you still paid for them. Context management helps future turns, not the original call that generated the waste.

## How AI Agent Memory Works and Why It Also Costs Tokens

Search isn't the only source of token overhead. Every piece of context an agent recalls from memory must also be loaded into the LLM's context window, and that costs tokens as well.

The LLM APIs that most agents rely on today are stateless: Anthropic's Messages API requires the full conversation history with every request, and OpenAI's Chat Completions API works the same way. Even OpenAI's newer stateful Responses API, which manages conversation state server-side, still bills for the full context window on every call. Memory loaded into context costs tokens regardless of how it gets there.

To work around this, agent frameworks write notes to files on disk and load relevant notes back into the context window when the agent needs them. For instance, OpenClaw stores curated notes in MEMORY.md and appends daily logs to timestamped Markdown files, then indexes them with hybrid BM25 and vector search so the agent can recall relevant context on demand.

OpenClaw's memory design works well, but it requires the full OpenClaw ecosystem: the Gateway process, messaging platform connections, and the rest of the stack. The same is true of Claude Code's memory, which is tied to its CLI. If you're building a custom agent outside these platforms, you need a standalone solution. The next section covers the tools available for both problems.

## How to Stop OpenClaw From Burning Through Tokens

If you want to reduce how many tokens OpenClaw consumes, there are two levers you can pull.

-   The first is **better retrieval** — replacing grep-style keyword dumps with ranked, relevance-driven search tools so the model only sees the information that actually matters.
-   The second is **better memory** — moving from opaque, framework-dependent storage to something you can understand, inspect, and control.

### Replacing grep with Better Retrieval: index1, QMD, and Milvus

Many AI coding agents search codebases with grep or ripgrep. Claude Code has a dedicated Grep tool built on ripgrep. OpenClaw doesn't have a built-in codebase search tool, but its exec tool lets the underlying model run any command, and skills like ripgrep or QMD can be loaded to guide how the agent searches. Without a retrieval-focused skill, the agent falls back on whatever approach the underlying model chooses. The core problem is the same across agents: without ranked retrieval, keyword matches enter the context window unfiltered.

This works when a project is small enough that every match fits comfortably in the context window. The problem starts when a codebase or document library grows to the point where a keyword returns dozens or hundreds of hits and the agent has to load all of them into the prompt. At that scale, you need results ranked by relevance, not just filtered by match.

The standard fix is hybrid search, which combines two complementary ranking methods:

-   BM25 scores each result by how often and how uniquely a term appears in a given document. A focused file that mentions "authentication" 15 times ranks higher than a sprawling file that mentions it once.
-   Vector search converts text into numerical representations of meaning, so "authentication" can match "login flow" or "session management" even when they share no keywords.

Neither method alone is sufficient: BM25 misses paraphrased terms, and vector search misses exact terms like error codes. Combining both and merging the ranked lists through a fusion algorithm covers both gaps.

The tools below implement this pattern at different scales. Grep is the baseline everyone starts with. index1, QMD, and Milvus each add hybrid search with increasing capacity.

#### index1: fast hybrid search on a single machine

[index1](https://github.com/gladego/index1) is a CLI tool that packages hybrid search into a single SQLite database file. FTS5 handles BM25, sqlite-vec handles vector similarity, and RRF fuses the ranked lists. Embeddings are generated locally by Ollama, so nothing leaves your machine.

index1 chunks code by structure, not by line count: Markdown files split by headings, Python files by AST, JavaScript and TypeScript by regex patterns. This means search results return coherent units like a full function or a complete documentation section, not arbitrary line ranges that cut off mid-block. Response time is 40 to 180ms for hybrid queries. Without Ollama, it falls back to BM25-only, which still ranks results rather than dumping every match into the context window.

index1 also includes an episodic memory module for storing lessons learned, bug root causes, and architectural decisions. These memories live inside the same SQLite database as the code index rather than as standalone files.

Note: index1 is an early-stage project (0 stars, 4 commits as of February 2026). Evaluate it against your own codebase before committing.

-   **Best for**: solo developers or small teams with a codebase that fits on one machine, looking for a fast improvement over grep.
-   **Outgrow it when**: you need multi-user access to the same index, or your data exceeds what a single SQLite file handles comfortably.

#### QMD: higher accuracy through local LLM re-ranking

[QMD](https://github.com/tobi/qmd) (Query Markup Documents), built by Shopify founder Tobi Lütke, adds a third stage: LLM re-ranking. After BM25 and vector search each return candidates, a local language model re-reads the top results and reorders them by actual relevance to your query. This catches cases where both keyword and semantic matches return plausible but wrong results.

QMD runs entirely on your machine using three GGUF models totaling about 2 GB: an embedding model (embeddinggemma-300M), a cross-encoder reranker (Qwen3-Reranker-0.6B), and a query expansion model (qmd-query-expansion-1.7B). All three download automatically on first run. No cloud API calls, no API keys.

The tradeoff is cold-start time: loading three models from disk takes roughly 15 to 16 seconds. QMD supports a persistent server mode (qmd mcp) that keeps models in memory between requests, eliminating the cold-start penalty for repeated queries.

-   **Best for:** privacy-critical environments where no data can leave your machine, and where retrieval accuracy matters more than response time.
-   **Outgrow it when:** you need sub-second responses, shared team access, or your dataset exceeds single-machine capacity.

#### Milvus: hybrid search at team and enterprise scale

The single-machine tools above work well for individual developers, but they hit limits when multiple people or agents need access to the same knowledge base. [](https://github.com/milvus-io/milvus)

[Milvus](https://github.com/milvus-io/milvus) is an open-source vector database built for that next stage: distributed, multi-user, and capable of handling billions of vectors.

Its key feature for this use case is built-in Sparse-BM25, available since Milvus 2.5 and significantly faster in 2.6. You provide raw text, and Milvus tokenizes it internally using an analyzer built on tantivy, then converts the result to sparse vectors that are pre-computed and stored at index time.

Because the BM25 representation is already stored, retrieval doesn't need to recalculate scores on the fly. These sparse vectors live alongside dense vectors (semantic embeddings) in the same Collection. At query time, you fuse both signals with a ranker such as RRFRanker, which Milvus provides out of the box. Same hybrid search pattern as index1 and QMD, but running on infrastructure that scales horizontally.

Milvus also provides capabilities that single-machine tools cannot: multi-tenant isolation (separate databases or collections per team), data replication with automatic failover, and hot/cold data tiering for cost-efficient storage. For agents, this means multiple developers or multiple agent instances can query the same knowledge base concurrently without stepping on each other's data.

-   **Best for**: multiple developers or agents sharing a knowledge base, large or fast-growing document sets, or production environments that need replication, failover, and access control.

To sum up:

| Tool | Stage | Deployment | Migration signal |
| --- | --- | --- | --- |
| Claude Native Grep | Prototyping | Built-in, zero setup | Bills climb or queries slow down |
| index1 | Single-machine (speed) | Local SQLite + Ollama | Need multi-user access or data outgrows one machine |
| QMD | Single-machine (accuracy) | Three local GGUF models | Need team-shared indexes |
| Milvus | Team or Production | Distributed cluster | Large document sets or multi-tenant requirements |

### Reducing AI Agent Token Costs by Giving Them Persistent, Editable Memory with memsearch

Search optimization reduces token waste per query, but it doesn't help with what the agent retains between sessions.

Every piece of context an agent recalls from memory has to be loaded into the prompt, and that costs tokens too. The question isn't whether to store memory, but how. The storage method determines whether you can see what the agent remembers, fix it when it's wrong, and take it with you if you switch tools.

Most frameworks fail on all three counts. Mem0 and Zep store everything in a vector database, which works for retrieval, but makes memory:

-   **Opaque.** You can't see what the agent remembers without querying an API.
-   **Hard to edit.** Correcting or removing a memory means API calls, not opening a file.
-   **Locked in.** Switching frameworks means exporting, converting, and reimporting your data.

OpenClaw takes a different approach. All memory lives in plain Markdown files on disk. The agent writes daily logs automatically, and humans can open and edit any memory file directly. This solves all three problems: the memory is readable, editable, and portable by design.

The trade-off is deployment overhead. Running OpenClaw's memory means running the full OpenClaw ecosystem: the Gateway process, messaging platform connections, and the rest of the stack. For teams already using OpenClaw, that's fine. For everyone else, the barrier is too high. **memsearch** was built to close this gap: it extracts OpenClaw's Markdown-first memory pattern into a standalone library that works with any agent.

**[memsearch](https://github.com/zilliztech/memsearch)**, built by Zilliz (the team behind Milvus), treats Markdown files as the single source of truth. A MEMORY.md holds long-term facts and decisions you write by hand. Daily logs (2026-02-26.md) are generated automatically from session summaries. The vector index, stored in Milvus, is a derived layer that can be rebuilt from the Markdown at any time.

In practice, this means you can open any memory file in a text editor, read exactly what the agent knows, and change it. Save the file, and memsearch's file watcher detects the change and re-indexes automatically. You can manage memories with Git, review AI-generated memories through pull requests, or move to a new machine by copying a folder. If the Milvus index is lost, you rebuild it from the files. The files are never at risk.

Under the hood, memsearch uses the same hybrid search pattern described above: chunks split by heading structure and paragraph boundaries, BM25 + vector retrieval, and an LLM-powered compact command that summarizes old memories when logs grow large.![](https://assets.zilliz.com/Blog_Open_Claw_Burning_Through_Tokens_3_d9df026b47.png)

Best for: teams that want full visibility into what the agent remembers, need version control over memory, or want a memory system that isn't locked to any single agent framework.

To sum up:

| Capability | Mem0 / Zep | memsearch |
| --- | --- | --- |
| Source of truth | Vector database (sole data source) | Markdown files (primary) + Milvus (index) |
| Transparency | Black box, requires API to inspect | Open any .md file to read |
| Editability | Modify via API calls | Edit directly in any text editor, auto re-indexed |
| Version control | Requires separate audit logging | Git works natively |
| Migration cost | Export → convert format → re-import | Copy the Markdown folder |
| Human-AI collaboration | AI writes, humans observe | Humans can edit, supplement, and review |

## Which setup fits your scale

| Scenario | Search | Memory | When to move on |
| --- | --- | --- | --- |
| Early prototype | Grep (built-in) | — | Bills climb or queries slow down |
| Single developer, search only | [index1](https://github.com/gladego/index1) (speed) or [QMD](https://github.com/tobi/qmd) (accuracy) | — | Need multi-user access or data outgrows one machine |
| Single developer, both | [index1](https://github.com/gladego/index1) | [memsearch](https://github.com/zilliztech/memsearch) | Need multi-user access or data outgrows one machine |
| Team or production, both | [Milvus](https://github.com/milvus-io/milvus) | [memsearch](https://github.com/zilliztech/memsearch) | — |
| Quick integration, memory only | — | Mem0 or Zep | Need to inspect, edit, or migrate memories |

## Conclusion

The token costs that come with always-on AI agents aren't inevitable. This guide covered two areas where better tooling can cut the waste: search and memory.

Grep works at small scale, but as codebases grow, unranked keyword matches flood the context window with content the model never needed. [](https://github.com/gladego/index1)[index1](https://github.com/gladego/index1) and [](https://github.com/tobi/qmd)[QMD](https://github.com/tobi/qmd) solve this on a single machine by combining BM25 keyword scoring with vector search and returning only the most relevant results. For teams, multi-agent setups, or production workloads, [](https://milvus.io)[Milvus](https://milvus.io) provides the same hybrid search pattern on infrastructure that scales horizontally.

For memory, most frameworks store everything in a vector database: opaque, hard to edit by hand, and locked to the framework that created it. [memsearch](https://github.com/zilliztech/memsearch) takes a different approach. Memory lives in plain Markdown files you can read, edit, and version-control with Git. Milvus serves as a derived index that can be rebuilt from those files at any time. You stay in control of what the agent knows.

Both [](https://github.com/zilliztech/memsearch)[memsearch](https://github.com/zilliztech/memsearch) and [](https://github.com/milvus-io/milvus)[Milvus](https://github.com/milvus-io/milvus) are open source. We're actively developing memsearch and would love feedback from anyone running it in production. Open an issue, submit a PR, or just tell us what's working and what isn't.

Projects mentioned in this guide:

-   [memsearch](https://github.com/zilliztech/memsearch): Markdown-first memory for AI agents, backed by Milvus.
-   [Milvus](https://github.com/milvus-io/milvus): Open-source vector database for scalable hybrid search.
-   [index1](https://github.com/gladego/index1): BM25 + vector hybrid search for AI coding agents.
-   [QMD](https://github.com/tobi/qmd): Local hybrid search with LLM re-ranking.

## Keep Reading

-   [We Extracted OpenClaw's Memory System and Open-Sourced It (memsearch)](https://milvus.io/blog/we-extracted-openclaws-memory-system-and-opensourced-it-memsearch.md)
-   [Persistent Memory for Claude Code: memsearch ccplugin](https://milvus.io/blog/adding-persistent-memory-to-claude-code-with-the-lightweight-memsearch-plugin.md)
-   [What Is OpenClaw? Complete Guide to the Open-Source AI Agent](https://milvus.io/blog/openclaw-formerly-clawdbot-moltbot-explained-a-complete-guide-to-the-autonomous-ai-agent.md)
-   [OpenClaw Tutorial: Connect to Slack for Local AI Assistant](https://milvus.io/blog/stepbystep-guide-to-setting-up-openclaw-previously-clawdbotmoltbot-with-slack.md)


Why AI Agents like OpenClaw Burn Through Tokens and How to Cut Costs



I recently used AI models to review a pull request, and the results were contradictory: Claude flagged a data race, while Gemini said the code was clean. That got me curious about how other AI models would behave, so I ran the latest flagship models from Claude, Gemini, Codex, Qwen, and MiniMax through a structured code-review benchmark. The results? The best-performing model caught only 53% of known bugs. 

However, my curiosity didn’t end there: what if these AI models worked together? I experimented with having them debate each other, and after five rounds of adversarial debate, bug detection jumped to 80%. The hardest bugs, ones requiring system-level understanding, hit 100% detection in debate mode.

This post walks through the experiment design, per-model results, and what the debate mechanism reveals about how to actually use AI for code review.

## Benchmarking Claude, Gemini, Codex, Qwen, and MiniMax for code review

If you’ve been using models for code review, you’ve probably noticed they don’t just differ in accuracy; they differ in how they read code. For example:

Claude usually walks the call chain top-to-bottom and will spend time on “boring” paths (error handling, retries, cleanup). That’s often where the real bugs hide, so I don’t hate the thoroughness.

Gemini tends to start with a strong verdict (“this is bad” / “looks fine”) and then works backwards to justify it from a design/structure angle. Sometimes that’s useful. Sometimes it reads like it skimmed and then committed to a take.

Codex is quieter. But when it flags something, it’s often concrete and actionable — less commentary, more “this line is wrong because X.”

These are impressions, though, not measurements. To get actual numbers, I set up a benchmark.

### Setup

**Five flagship models were tested:**

-   Claude Opus 4.6
    
-   Gemini 3 Pro
    
-   GPT-5.2-Codex
    
-   Qwen-3.5-Plus
    
-   MiniMax-M2.5
    

**Tooling (Magpie)**

I used [Magpie](https://github.com/liliu-z/magpie), an open-source benchmarking tool I built. Its job is to do the “code review prep” you’d normally do manually: pull in surrounding context (call chains, related modules, and relevant adjacent code) and feed that to the model *before* it reviews the PR.

**Test cases (Milvus PRs with known bugs)**

The dataset consists of 15 pull requests from [Milvus](https://github.com/milvus-io/milvus) (an open-source vector database created and maintained by [Zilliz](https://zilliz.com/)). These PRs are useful as a benchmark because each was merged, only to later require a revert or hotfix after a bug surfaced in production. So every case has a known bug we can score against.

**Bug difficulty levels**

Not all of these bugs are equally hard to find, though, so I categorized them into three difficulty levels:

-   **L1:** Visible from the diff alone (use-after-free, off-by-one). 
    
-   **L2 (10 cases):** Requires understanding surrounding code to spot things like interface semantic changes or concurrency races. These represent the most common bugs in daily code review.
    
-   **L3 (5 cases):** Requires system-level understanding to catch issues like cross-module state inconsistencies or upgrade compatibility problems. These are the hardest tests of how deeply a model can reason about a codebase.
    

*Note: Every model caught all L1 bugs, so I excluded them from scoring.*

**Two evaluation modes**

Each model was run in two modes:

-   **Raw:** the model sees only the PR (diff + whatever is in the PR content).
    
-   **R1:** Magpie pulls the surrounding context (relevant files / call sites / related code) *before* the model reviews. This simulates a workflow where you prep context up front instead of asking the model to guess what it needs.
    

### Results (L2 + L3 only)

| Mode | Claude | Gemini | Codex | MiniMax | Qwen |
| --- | --- | --- | --- | --- | --- |
| Raw | 53% (1st) | 13% (last) | 33% | 27% | 33% |
| R1 (with context by Magpie) | 47% ⬇️ | 33%⬆️ | 27% | 33% | 40%⬆️ |

Four takeaways:

**1. Claude dominates the raw review.** It scored 53% overall detection and a perfect 5/5 on L3 bugs, without any context assistance. If you're using a single model and don't want to spend time preparing context, Claude is the best choice.

**2. Gemini needs context handed to it.** Its raw score of 13% was the lowest in the group, but with Magpie providing surrounding code, it jumped to 33%. Gemini doesn't gather its own context well, but it performs respectably when you do that work upfront.

**3. Qwen is the strongest context-assisted performer.** It scored 40% in R1 mode, with 5/10 on L2 bugs, which was the highest score at that difficulty level. For routine daily reviews where you're willing to prepare context, Qwen is a practical pick.

**4. More context doesn't always help.** It lifted Gemini (13% → 33%) and MiniMax (27% → 33%), but it actually hurt Claude (53% → 47%). Claude already excels at organizing context on its own, so the additional information likely introduced noise rather than clarity. The lesson: match the workflow to the model, rather than assuming more context is universally better.

These results align with my day-to-day experience. Claude at the top isn't surprising. Gemini scoring lower than I expected makes sense in hindsight: I typically use Gemini in multi-turn conversations where I'm iterating on a design or chasing a problem together, and it performs well in that interactive setting. This benchmark is a fixed, single-pass pipeline, which is exactly the format in which Gemini is weakest. The debate section later will show that when you give Gemini a multi-round, adversarial format, its performance improves noticeably.

## Let AI Models Debate with Each Other 

Every model showed different strengths and blind spots in the individual benchmarks. So I wanted to test: what happens if the models review each other's work rather than just the code?

So I added a debate layer on top of the same benchmark. All five models participate in five rounds:

-   In Round 1, each model reviews the same PR independently.
    
-   After that, I broadcast all five reviews to all participants.
    
-   In Round 2, each model updates its position based on the other four.
    
-   Repeat until Round 5.
    

By the end, each model isn’t just reacting to the code — it’s reacting to arguments that have already been criticized and revised multiple times.

To keep this from turning into “LLMs agreeing loudly,” I enforced one hard rule: **every claim has to point to specific code as evidence**, and a model can’t just say “good point” — it has to explain why it changed its mind.

### Results: Best Solo vs Debate Mode

| Mode | L2 (10 cases) | L3 (5 cases) | Total detection |
| --- | --- | --- | --- |
| Best individual (Raw Claude) | 3/10 | 5/5 | 53% |
| Debate (all five models) | 7/10 (doubled) | 5/5 (all caught) | 80% |

### What stands out

**1. L2 detection doubled.** Routine, mid-difficulty bugs jumped from 3/10 to 7/10. These are the bugs that appear most frequently in real codebases, and they're exactly the category where individual models miss inconsistently. The debate mechanism's biggest contribution is closing these everyday gaps.

**2. L3 bugs: zero misses.** In the single-model runs,  only Claude caught all five L3 system-level bugs. In debate mode, the group matched that result, meaning you no longer need to bet on the right model to get full L3 coverage.

**3. Debate fills blind spots rather than raising the ceiling.** System-level bugs weren't the hard part for the strongest individual. Claude already had those. The debate mechanism's core contribution is patching Claude's weakness on routine L2 bugs, where individual Claude caught only 3 out of 10, but the debating group caught 7. That's where the 53% → 80% jump comes from.

### What debate actually looks like in practice

The numbers above show that debate works, but a concrete example shows *why* it works. Here's a condensed walkthrough of how the five models handled **PR #44474**, which introduced lazy loading of primary keys during search.

[**PR #44474:**](https://github.com/milvus-io/milvus/pull/44474) A lazy-loading optimization that changed search to fetch primary keys on demand instead of loading them all upfront.

**Round 1**

Gemini opened aggressively:

"This PR is a minefield of bad practices, potential undefined behavior, and performance regressions disguised as an 'optimization.'"

Claude and Qwen focused on undefined behavior and performance, converging on the same concern:

"The C++ memory model doesn't guarantee safety for concurrent writes to different indices of the same vector without proper synchronization."

All three found real issues, but none of them caught the deeper bug. Then Codex, which had barely spoken, flagged something the others missed: the `primary_keys_` vector gets resized, but only positions that pass through the reduce phase are actually populated. Every other position is left as a zero value.

**Round 2**

Claude picked up Codex's finding and traced the downstream consequence:

"I found it: `SortEqualScoresByPks` runs before `ReduceResultData`. So when `SortEqualScoresByPks` executes, zero PKs have been lazily fetched yet. The entire `primary_keys_` vector is full of default-constructed PkType values."

In plain terms, the sort function runs before the keys have been loaded, so it's sorting zeros. Claude acknowledged the miss openly:

"codex-cli identified a genuine critical correctness bug. I missed this in my first round."

## Which Combination of Models Can Find the Most Bugs?

The jump from 53% to 80% happened because five models covered each other's blind spots. But not everyone can afford to set up and run five models through five rounds of debate for every code review. 

**So I tested the simpler version: if you can only run two models, which pair gets you closest to the multi-model ceiling?**

I used the **context-assisted (R1)** runs and counted how many of the 15 known bugs each model found:

-   **Claude:** 7/15 (47%)
    
-   **Qwen:** 6/15 (40%)
    
-   **Gemini:** 5/15 (33%)
    
-   **MiniMax:** 5/15 (33%)
    
-   **Codex:** 4/15 (27%)
    

What matters, then, is not just how many bugs each model finds, but *which* bugs it misses. Of the 8 bugs Claude missed, Gemini caught 3: a concurrency race condition, a cloud storage API compatibility issue, and a missing permission check. Going the other direction, Gemini missed most data structures and deep logic bugs, and Claude caught nearly all of those. Their weaknesses barely overlap, which is what makes them a strong pair.

  

| Two-model pairing | Combined coverage |
| --- | --- |
| Claude + Gemini | 10/15 |
| Claude + Qwen | 9/15 |
| Claude + Codex | 8/15 |
| Claude + MiniMax | 8/15 |

  

All five models together covered 11 out of 15, leaving 4 bugs that every model missed. 

**Claude + Gemini,** as a two-model pair, already reaches 91% of that five-model ceiling. For this benchmark, it's the most efficient combination.

That said, Claude + Gemini isn't the best pairing for every type of bug. When I broke the results down by bug category, a more nuanced picture emerged:

| Bug type | Total | Claude | Gemini | Codex | MiniMax | Qwen |
| --- | --- | --- | --- | --- | --- | --- |
| Validation gaps | 4 | 3 | 2 | 1 | 1 | 3 |
| Data structure lifecycle | 4 | 3 | 1 | 1 | 3 | 1 |
| Concurrency races | 2 | 0 | 1 | 0 | 0 | 0 |
| Compatibility | 2 | 0 | 1 | 1 | 0 | 1 |
| Deep logic | 3 | 1 | 0 | 1 | 1 | 1 |
| Total | 15 | 7 | 5 | 4 | 5 | 6 |

The bug-type breakdown reveals why no single pairing is universally best. 

-   For data structure lifecycle bugs, Claude and MiniMax tied at 3/4. 
    
-   For validation gaps, Claude and Qwen tied at 3/4. 
    
-   For concurrency and compatibility issues, Claude scored zero on both, and Gemini is the one that fills those gaps. 
    
-   No model covers everything, but Claude covers the widest range and comes closest to being a generalist.
    

Four bugs were missed by every model. One involved ANTLR grammar rule priority. One was a read/write lock semantic mismatch across functions. One required understanding the business logic differences between compaction types. And one was a silent comparison error where one variable used megabytes and another used bytes.

What these four have in common is that the code is syntactically correct. The bugs live in assumptions the developer carried in their head, not in the diff, and not even in the surrounding code. This is roughly where AI code review hits its ceiling today.

## After Finding Bugs, Which Model is the Best at Fixing Them?

In code review, finding bugs is half the job. The other half is fixing them. So after the debate rounds, I added a peer evaluation to measure how useful each model's fix suggestions actually are.

To measure this, I added a peer evaluation round after the debate. Each model opened a fresh session and acted as an anonymous judge, scoring the other models' reviews. The five models were randomly mapped to Reviewer A/B/C/D/E, so no judge knew which model produced which review. Each judge scored on four dimensions, rated 1 to 10: accuracy, actionability, depth, and clarity.

  

| Model | Accuracy | Actionability | Depth | Clarity | Overall |
| --- | --- | --- | --- | --- | --- |
| Qwen | 8.6 | 8.6 | 8.5 | 8.7 | 8.6 (tied 1st) |
| Claude | 8.4 | 8.2 | 8.8 | 8.8 | 8.6 (tied 1st) |
| Codex | 7.7 | 7.6 | 7.1 | 7.8 | 7.5 |
| Gemini | 7.4 | 7.2 | 6.7 | 7.6 | 7.2 |
| MiniMax | 7.1 | 6.7 | 6.9 | 7.4 | 7.0 |

  

Qwen and Claude tied for first by a clear margin. Both scored consistently high across all four dimensions, while Codex, Gemini, and MiniMax clustered a full point or more below. Notably, Gemini, which proved valuable as a bug-finding partner for Claude in the pairing analysis, ranks near the bottom for review quality. Being good at spotting issues and being good at explaining how to fix them are evidently different skills.

  

## Conclusion

**Claude** is the one you'd trust with the hardest reviews. It works through entire call chains, follows deep logic paths, and pulls in its own context without you needing to spoon-feed it. On L3 system-level bugs, nothing else came close. It does get overconfident with math sometimes, but when another model proves it wrong, it owns it and walks through where its reasoning broke down. Use it for core code and the bugs you can't afford to miss.

**Gemini** comes in hot. It has strong opinions about code style and engineering standards, and it's quick to frame problems structurally. The downside is that it often stays at the surface and doesn't dig deep enough, which is why it scored low in peer evaluation. Where Gemini really earns its spot is as a challenger: its pushback forces other models to double-check their work. Pair it with Claude for the structural perspective Claude sometimes skips.

**Codex** barely says a word. But when it does, it counts. Its hit rate on real bugs is high, and it has a knack for catching the one thing everyone else walked past. In the PR #44474 example, Codex was the model that spotted the zero-value primary keys issue that kicked off the whole chain. Think of it as the supplementary reviewer that catches what your primary model missed.

**Qwen** is the most well-rounded of the five. Its review quality matched Claude's, and it's especially good at pulling together different perspectives into fix suggestions you can actually act on. It also had the highest L2 detection rate in context-assisted mode, which makes it a solid default for everyday PR reviews. The one weakness: in long, multi-round debates, it sometimes loses track of earlier context and starts giving inconsistent answers in later rounds.

**MiniMax** was the weakest at finding bugs on its own. It's best used to fill out a multi-model group rather than as a standalone reviewer.

## Limitations of This Experiment

A few caveats to keep this experiment in perspective:

**The sample size is small.** There are only 15 PRs, all from the same Go/C++ project (Milvus). These results don't generalize to all languages or codebases. Treat them as directional, not definitive.

**Models are inherently random.** Running the same prompt twice can produce different results. The numbers in this post are a single snapshot, not a stable expected value. The individual model rankings should be taken lightly, though the broader trends (debate outperforms individuals, different models excel at different bug types) are consistent.

**The speaking order was fixed.** The debate used the same order across all rounds, which may have influenced how later-speaking models responded. A future experiment could randomize the order per round to control for this.

## Try it yourself

All tools and data from this experiment are open source:

-   [**Magpie**](https://github.com/liliu-z/magpie): An open-source tool that gathers code context (call chains, related PRs, affected modules) and orchestrates multi-model adversarial debate for code review.
    
-   [**AI-CodeReview-Arena**](https://github.com/liliu-z/ai-code-review-arena): The full evaluation pipeline, configurations, and scripts.
    
-   [**Test cases**](https://github.com/liliu-z/ai-code-review-arena/blob/main/prs/manifest.yaml): All 15 PRs with annotated known bugs.
    

The bugs in this experiment all came from real pull requests in [Milvus](https://github.com/milvus-io/milvus), an open-source vector database built for AI applications. We've got a pretty active community on [Discord](https://discord.com/invite/8uyFbECzPX) and [Slack](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email), and we'd love more people poking at the code. And if you end up running this benchmark on your own codebase, please share the results! I'm really curious whether the trends hold across different languages and projects.

## Keep Reading

-   [GLM-5 vs. MiniMax M2.5 vs. Gemini 3 Deep Think: Which Model Fits Your AI Agent Stack?](https://milvus.io/blog/glm5-vs-minimax-m25-vs-gemini-3-deep-think.md)
    
-   [Adding Persistent Memory to Claude Code with the Lightweight memsearch Plugin](https://milvus.io/blog/adding-persistent-memory-to-claude-code-with-the-lightweight-memsearch-plugin.md)
    
-   [We Extracted OpenClaw’s Memory System and Open-Sourced It (memsearch)](https://milvus.io/blog/we-extracted-openclaws-memory-system-and-opensourced-it-memsearch.md)


Claude vs Gemini vs Codex vs Qwen vs MiniMax Code Review


AI Code Review Gets Better When Models Debate: Claude vs Gemini vs Codex vs Qwen vs MiniMax


Production AI Agents with Persistent Memory Using Google ADK and Milvus


How to Build Production-Ready AI Agents with Long-Term Memory Using Google ADK and Milvus


Debugging RAG in 3D with Project_Golem and Milvus


What If You Could See Why RAG Fails? Debugging RAG in 3D with Project_Golem and Milvus


GLM-5 vs MiniMax M2.5 vs Gemini 3 Deep Think Compared


GLM-5 vs. MiniMax M2.5 vs. Gemini 3 Deep Think: Which Model Fits Your AI Agent Stack?





We recently built and open-sourced [memsearch](https://github.com/zilliztech/memsearch), a standalone, plug-and-play long-term memory library that gives any agent persistent, transparent, and human-editable memory. It uses the same underlying memory architecture as OpenClaw—just without the rest of the OpenClaw stack. That means you can drop it into any agent framework (Claude, GPT, Llama, custom agents, workflow engines) and instantly add durable, queryable memory. *(If you want the deep dive into how memsearch works, we wrote a* [*separate post here*](https://milvus.io/blog/we-extracted-openclaws-memory-system-and-opensourced-it-memsearch.md)*.)*

In most agent workflows, memsearch performs exactly as intended. But **agentic coding** is a different story. Coding sessions run long, context switches are constant, and the information worth keeping accumulates over days or weeks. That sheer volume and volatility expose weaknesses in typical agent memory systems—memsearch included. In coding scenarios, retrieval patterns differ enough that we couldn’t simply reuse the existing tool as-is.

To address this, we built a **persistent memory plugin designed specifically for Claude Code**. It sits on top of the memsearch CLI, and we’re calling it the **memsearch ccplugin**.

-   GitHub Repo: [](https://github.com/zilliztech/memsearch)[https://zilliztech.github.io/memsearch/claude-plugin/](https://zilliztech.github.io/memsearch/claude-plugin/) *(open-source, MIT license)*
    

With the lightweight **memsearch ccplugin** managing memory behind the scenes, Claude Code gains the ability to remember every conversation, every decision, every style preference, and every multi-day thread—automatically indexed, fully searchable, and persistent across sessions.

![](https://assets.zilliz.com/memsearch_plugin_diagram_41563f84dd.png)

*For clarity throughout this post: "ccplugin" refers to the upper layer, or the Claude Code plugin itself. "memsearch" refers to the lower layer, the standalone CLI tool underneath it.*

So why does coding need its own plugin, and why did we build something so lightweight? It comes down to two problems you've almost certainly hit: Claude Code's lack of persistent memory, and the clunkiness and complexity of existing solutions like claude-mem.

So why build a dedicated plugin at all? Because coding agents run into two pain points you’ve almost certainly experienced yourself: 

-   Claude Code has no persistent memory.
    
-   Many existing community solutions—like *claude-mem*—are powerful but heavy, clunky, or overly complex for day-to-day coding work. 
    

The ccplugin aims to solve both problems with a minimal, transparent, developer-friendly layer on top of memsearch.

## Claude Code's Memory Problem: It Forgets Everything When a Session Ends

Let’s start with a scenario that Claude Code Users most definitely have run into.

You open Claude Code in the morning. "Continue yesterday's auth refactor," you type. Claude replies: "I'm not sure what you were working on yesterday." So you spend the next ten minutes copy-pasting yesterday’s logs. It’s not a huge problem, but it gets annoying quickly because it appears so frequently. 

Even though Claude Code has its own memory mechanisms, they’re far from satisfactory. The `CLAUDE.md` file can store project directives and preferences, but it works better for static rules and short commands, not for accumulating long-term knowledge.

Claude Code does offer `resume` and `fork` commands, but they're far from user-friendly. For fork commands, you need to remember session IDs, type commands manually, and manage a tree of branching conversation histories. When you run `/resume`, you get a wall of session titles. If you only remember a few details about what you did and it was more than a few days ago, good luck finding the right one.

![](https://assets.zilliz.com/code_snippet_82ec01cc5e.jpeg)

For long-term, cross-project knowledge accumulation, this whole approach is impossible. 

To deliver on that idea, claude-mem uses a three-tier memory system. The first tier searches high-level summaries. The second tier digs into a timeline for more detail. The third tier pulls full observations for raw conversation. On top of that, there are privacy labels, cost tracking, and a web visualization interface.

Here's how it works under the hood:

-   **Runtime layer.** A Node.js Worker service runs on port 37777. Session metadata lives in a lightweight SQLite database. A vector database handles precise semantic retrieval over memory content.
    
-   **Interaction layer.** A React-based web UI lets you view captured memories in real time: summaries, timelines, and raw records.
    
-   **Interface layer.** An MCP (Model Context Protocol) server exposes standardized tool interfaces. Claude can call `search` (query high-level summaries), `timeline` (view detailed timelines), and `get_observations` (retrieve raw interaction records) to retrieve and use memories directly.
    

To be fair, this is a solid product that solves Claude Code's memory problem. But it's clunky and complex in ways that matter day-to-day.

| Layer | Technology |
| --- | --- |
| Language | TypeScript (ES2022, ESNext modules) |
| Runtime | Node.js 18+ |
| Database | SQLite 3 with bun:sqlite driver |
| Vector Store | ChromaDB (optional, for semantic search) |
| HTTP Server | Express.js 4.18 |
| Real-time | Server-Sent Events (SSE) |
| UI Framework | React + TypeScript |
| AI SDK | @anthropic-ai/claude-agent-sdk |
| Build Tool | esbuild (bundles TypeScript) |
| Process Manager | Bun |
| Testing | Node.js built-in test runner |



**For starters, setup is heavy.** Getting claude-mem running means installing Node.js, Bun, and the MCP runtime, then standing up a Worker service, Express server, React UI, SQLite, and a vector store on top of that. That's a lot of moving parts to deploy, maintain, and debug when something breaks.

**All those components also burn tokens you didn't ask to spend.** MCP tool definitions load permanently into Claude's context window, and every tool call eats tokens on the request and response. Over long sessions, that overhead adds up fast and can push token costs out of control.

**Memory recall is unreliable because it depends entirely on Claude choosing to search.** Claude has to decide on its own to call tools like `search` to trigger retrieval. If it doesn't realize it needs a memory, the relevant content just never shows up. And each of the three memory tiers requires its own explicit tool invocation, so there's no fallback if Claude doesn't think to look.

**Finally, data storage is opaque, which makes debugging and migration unpleasant.** Memories are split across SQLite for session metadata and Chroma for binary vector data, with no open format tying them together. Migrating means writing export scripts. Seeing what the AI actually remembers means going through the Web UI or a dedicated query interface. There's no way to just look at the raw data.

## Why the memsearch Plugin for Claude Code is Better?  

We wanted a memory layer that was truly lightweight—no extra services, no tangled architecture, no operational overhead. That’s what motivated us to build the **memsearch ccplugin**. At its core, this was an experiment: *could a coding-focused memory system be radically simpler?*

Yes, and we proved it. 

![](https://assets.zilliz.com/memsearch_icon_d68365006a.png)

The entire ccplugin is four shell hooks plus a background watch process. No Node.js, no MCP server, no Web UI. It's just shell scripts calling the memsearch CLI, which drops the setup and maintenance bar dramatically.

The ccplugin can be this thin because of strict responsibility boundaries. It doesn't handle memory storage, vector retrieval, or text embedding. All of that is delegated to the memsearch CLI underneath. The ccplugin has one job: bridge Claude Code's lifecycle events (session start, prompt submission, response stop, session end) to the corresponding memsearch CLI functions.

![](https://assets.zilliz.com/memsearch_diagram_2_6b2dbeaaf6.png)

This decoupled design makes the system flexible beyond Claude Code. The memsearch CLI works independently with other IDEs, other agent frameworks, or even plain manual invocation. It's not locked to a single use case. 

In practice, this design delivers three key advantages.

### 1. All Memories Live in Plain Markdown Files

Every memory the ccplugin creates lives in `.memsearch/memory/` as a Markdown file.

```
.memsearch/memory/
├── 2026-02-09.md
├── 2026-02-10.md
└── 2026-02-11.md
```

It’s one file per day. Each file contains that day's session summaries in plain text, fully human-readable. Here's a screenshot of the daily memory files from the memsearch project itself:

![](https://assets.zilliz.com/markdown_file_d0ab53e13b.jpeg)

You can see the format right away: timestamp, session ID, turn ID, and a summary of the session. Nothing is hidden.

Want to know what the AI remembers? Open the Markdown file. Want to edit a memory? Use your text editor. Want to migrate your data? Copy the `.memsearch/memory/` folder.

The [Milvus](https://milvus.io/) vector index is a cache to speed up semantic search. It rebuilds from Markdown at any time. No opaque databases, no binary black boxes. All data is traceable and fully reconstructable.

![](https://assets.zilliz.com/milvus_index_workflow_e8de4628da.png)

### 2. Automatic Context Injection Costs Zero Extra Tokens

Transparent storage is the foundation of this system. The real payoff comes from how these memories get used, and in ccplugin, memory recall is fully automatic.

Every time a prompt is submitted, the `UserPromptSubmit` hook fires a semantic search and injects the top-3 relevant memories into context. Claude doesn't decide whether to search. It just gets the context.

During this process, Claude never sees MCP tool definitions, so nothing extra occupies the context window. The hook runs at the CLI layer and injects plain text search results. No IPC overhead, no tool-call token costs. The context-window bloat that comes with MCP tool definitions is gone entirely.

![](https://assets.zilliz.com/diagram_3_b9e8391c2a.png)

For cases where the automatic top-3 isn't enough, we also built three tiers of progressive retrieval. All three are CLI commands, not MCP tools.

-   **L1 (automatic):** Every prompt returns the top-3 semantic search results with a `chunk_hash` and 200-character preview. This covers most everyday use.
    
-   **L2 (on-demand):** When full context is needed, `memsearch expand <chunk_hash>` returns the complete Markdown section plus metadata.
    
-   **L3 (deep):** When the original conversation is needed, `memsearch transcript <jsonl_path> --turn <uuid>` pulls the raw JSONL record from Claude Code.
    

![](https://assets.zilliz.com/memsearch_diagram_4_ccc495d5ac.png)

![](https://assets.zilliz.com/memsearch_diagram_5_0333650103.png)

  

### 3. Session Summaries Are Generated in the Background at Near-Zero Cost

Retrieval covers how memories get used. But the memories have to be written first. How do all those Markdown files get created?

The ccplugin generates them through a background pipeline that runs asynchronously and costs almost nothing. Every time you stop a Claude response, the `Stop` hook fires: it parses the conversation transcript, calls Claude Haiku (`claude -p --model haiku`) to generate a summary, and appends it to the current day's Markdown file. Haiku API calls are extremely cheap, nearly negligible per invocation.

From there, the watch process detects the file change and automatically indexes the new content into Milvus so it's available for retrieval right away. The whole flow runs in the background without interrupting your work, and costs stay controlled.

## Quickstart memsearch plugin with Claude Code 

### First, install from the Claude Code plugin marketplace:

```

bash
# Run in Claude Code terminal
/plugin marketplace add zilliztech/memsearch
/plugin install memsearch
```

### Second, restart Claude Code. 

The plugin initializes its configuration automatically.

### Third, after a conversation, check the day's memory file:

```
bash
cat .memsearch/memory/$(date +%Y-%m-%d).md
```

### Fourth, enjoy.

The next time Claude Code starts, the system automatically retrieves and injects relevant memories. No extra steps needed.

## Conclusion

Let’s go back to the original question: how do you give AI persistent memory? claude-mem and memsearch ccplugin take different approaches, each with different strengths. We summed up a quick guide to choosing between them:

  

| Category | memsearch | claude-mem |
| --- | --- | --- |
| Architecture | 4 shell hooks + 1 watch process | Node.js Worker + Express + React UI |
| Integration Method | Native hooks + CLI | MCP server (stdio) |
| Recall | Automatic (hook injection) | Agent-driven (requires tool invocation) |
| Context Consumption | Zero (inject result text only) | MCP tool definitions persist |
| Session Summary | One asynchronous Haiku CLI call | Multiple API calls + observation compression |
| Storage Format | Plain Markdown files | SQLite + Chroma embeddings |
| Data Migration | Plain Markdown files | SQLite + Chroma embeddings |
| Migration Method | Copying .md files | Exporting from database |
| Runtime | Python + Claude CLI | Node.js + Bun + MCP runtime |

  

claude-mem offers richer features, a polished UI, and finer-grained control. For teams that need collaboration, web visualization, or detailed memory management, it's a strong pick.

memsearch ccplugin offers minimal design, zero context-window overhead, and fully transparent storage. For engineers who want a lightweight memory layer without additional complexity, it's the better fit. Which one is better depends on what you need.

Want to dive deeper or get help building with memsearch or Milvus?

-   Join the [Milvus Slack community](https://milvus.io/slack) t to connect with other developers and share what you're building.
    
-   Book our [Milvus Office Hours](https://milvus.io/office-hours)for live Q&A and direct support from the team.
    

  

## Resources

-   **memsearch ccplugin documentation:** [https://zilliztech.github.io/memsearch/claude-plugin/](https://zilliztech.github.io/memsearch/claude-plugin/)
    
-   **GitHub:** [https://github.com/zilliztech/memsearch/tree/main/ccplugin](https://github.com/zilliztech/memsearch/tree/main/ccplugin)
    
-   **memsearch project:** [https://github.com/zilliztech/memsearch](https://github.com/zilliztech/memsearch) 
    
-   Blog: [We Extracted OpenClaw’s Memory System and Open-Sourced It (memsearch)](https://milvus.io/blog/we-extracted-openclaws-memory-system-and-opensourced-it-memsearch.md) 
    
-   Blog: [What Is OpenClaw? Complete Guide to the Open-Source AI Agent -](https://milvus.io/blog/openclaw-formerly-clawdbot-moltbot-explained-a-complete-guide-to-the-autonomous-ai-agent.md)
    
-   Blog: [OpenClaw Tutorial: Connect to Slack for Local AI Assistant](https://milvus.io/blog/stepbystep-guide-to-setting-up-openclaw-previously-clawdbotmoltbot-with-slack.md)


Persistent Memory for Claude Code: memsearch ccplugin


Adding Persistent Memory to Claude Code with the Lightweight memsearch Plugin


We Extracted OpenClaw’s Memory System and Open-Sourced It (memsearch)



If you've been building AI agents, you've probably hit this wall: your demo works great, but getting it into production is a whole different story.

We've covered agent memory management and reranking in earlier posts. Now let's tackle the bigger challenge—building agents that actually hold up in production.

Here's the reality: production environments are messy. A single agent rarely cuts it, which is why multi-agent systems are everywhere. But the frameworks available today tend to fall into two camps: lightweight ones that demo well but break under real load, or powerful ones that take forever to learn and build with.

I've been experimenting with [Agno](https://github.com/agno-agi/agno) recently, and it seems to strike a reasonable middle ground—focused on production readiness without excessive complexity. The project has gained over 37,000 GitHub stars in a few months, suggesting other developers find it useful as well.

In this post, I'll share what I learned while building a multi-agent system using Agno with [Milvus](https://milvus.io/) as the memory layer. We'll look at how Agno compares to alternatives such as LangGraph and walk through a complete implementation you can try yourself.

## What Is Agno?

[Agno](https://github.com/agno-agi/agno) is a multi-agent framework built specifically for production use. It has two distinct layers:

-   **Agno framework layer**: Where you define your agent logic
    
-   **AgentOS runtime layer**: Turns that logic into HTTP services you can actually deploy
    

Think of it this way: the framework layer defines *what* your agents should do, while AgentOS handles *how* that work gets executed and served.

### The Framework Layer

This is what you work with directly. It introduces three core concepts:

-   **Agent**: Handles a specific type of task
    
-   **Team**: Coordinates multiple agents to solve complex problems
    
-   **Workflow**: Defines execution order and structure
    

One thing I appreciated: you don't need to learn a new DSL or draw flowcharts. Agent behavior is defined using standard Python function calls. The framework handles LLM invocation, tool execution, and memory management.

### The AgentOS Runtime Layer

AgentOS is designed for high request volumes through async execution, and its stateless architecture makes scaling straightforward.

Key features include:

-   Built-in FastAPI integration for exposing agents as HTTP endpoints
    
-   Session management and streaming responses
    
-   Monitoring endpoints
    
-   Horizontal scaling support
    

In practice, AgentOS handles most of the infrastructure work, which lets you focus on the agent logic itself.

A high-level view of Agno’s architecture is shown below.

![](https://assets.zilliz.com/1_dfbf444ee6.png)

## Agno vs. LangGraph

To understand where Agno fits, let's compare it with LangGraph—one of the most widely used multi-agent frameworks.

[**LangGraph**](https://www.langchain.com/langgraph) uses a graph-based state machine. You model your entire agent workflow as a graph: steps are nodes, execution paths are edges. This works well when your process is fixed and strictly ordered. But for open-ended or conversational scenarios, it can feel restrictive. As interactions get more dynamic, maintaining a clean graph gets harder.

**Agno** takes a different approach. Instead of being a pure orchestration layer, it's an end-to-end system. Define your agent behavior, and AgentOS automatically exposes it as a production-ready HTTP service—with monitoring, scalability, and multi-turn conversation support built in. No separate API gateway, no custom session management, no extra operational tooling.

Here's a quick comparison:

| Dimension | LangGraph | Agno |
| --- | --- | --- |
| Orchestration model | Explicit graph definition using nodes and edges | Declarative workflows defined in Python |
| State management | Custom state classes defined and managed by developers | Built-in memory system |
| Debugging & observability | LangSmith (paid) | AgentOS UI (open source) |
| Runtime model | Integrated into an existing runtime | Standalone FastAPI-based service |
| Deployment complexity | Requires additional setup via LangServe | Works out of the box |

LangGraph gives you more flexibility and fine-grained control. Agno optimizes for faster time-to-production. The right choice depends on your project stage, existing infrastructure, and the level of customization you need. If you're unsure, running a small proof of concept with both is probably the most reliable way to decide.

## Choosing Milvus for the Agent Memory Layer

Once you've chosen a framework, the next decision is how to store memory and knowledge. We use Milvus for this. [Milvus](https://milvus.io/) is the most popular open-source vector database built for AI workloads with more than [42,000+ GitHub](https://github.com/milvus-io/milvus) stars. 

**Agno has native Milvus support.** The `agno.vectordb.milvus` module wraps production features like connection management, automatic retries, batch writes, and embedding generation. You don't need to build connection pools or handle network failures yourself—a few lines of Python give you a working vector memory layer.

**Milvus scales with your needs.** It supports three [deployment modes:](https://milvus.io/docs/install-overview.md)

-   **Milvus Lite**: Lightweight, file-based—great for local development and testing
    
-   **Standalone**: Single-server deployment for production workloads
    
-   **Distributed**: Full cluster for high-scale scenarios
    

You can start with Milvus Lite to validate your agent memory locally, then move to standalone or distributed as traffic grows—without changing your application code. This flexibility is especially useful when you're iterating quickly in early stages but need a clear path to scale later.

![](https://assets.zilliz.com/1_1_1en_e0294d0ffa.png)

## Step-by-Step: Building a Production-Ready Agno Agent with Milvus

Let's build a production-ready agent from scratch.

We'll start with a simple single-agent example to show the full workflow. Then we'll expand it into a multi-agent system. AgentOS will automatically package everything as a callable HTTP service.

### 1. Deploying Milvus Standalone with Docker

**(1) Download the Deployment Files**

```
**wget** **
<https://github.com/Milvus-io/Milvus/releases/download/v2.****5****.****12****/Milvus-standalone-docker-compose.yml> -O docker-compose.yml**
```

**(2) Start the Milvus Service**

```
docker-compose up -d
```

```
docker-compose ps -a
```

![](https://assets.zilliz.com/_80575354d3.png)

### 2. Core Implementation

```
import os
from pathlib import Path
from agno.os import AgentOS
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.knowledge.knowledge import Knowledge
from agno.vectordb.milvus import Milvus
from agno.knowledge.embedder.openai import OpenAIEmbedder
from agno.db.sqlite import SqliteDb
os.environ\["OPENAI_API_KEY"\] = "you-key-here"
data_dir = Path("./data")
data_dir.mkdir(exist_ok=True)
knowledge_base = Knowledge(
    contents_db=SqliteDb(
        db_file=str(data_dir / "knowledge_contents.db"),
        knowledge_table="knowledge_contents",
    ),
    vector_db=Milvus(
        collection="agno_knowledge",
        uri="http://192.168.x.x:19530",
        embedder=OpenAIEmbedder(id="text-embedding-3-small"),
    ),
)
*# Create Agent*
agent = Agent(
    name="Knowledge Assistant",
    model=OpenAIChat(id="gpt-4o"),
    instructions=\[
        "You are a knowledge base assistant that helps users query and manage knowledge base content.",
        "Answer questions in English.",
        "Always search the knowledge base before answering questions.",
        "If the knowledge base is empty, kindly prompt the user to upload documents."
    \],
    knowledge=knowledge_base,
    search_knowledge=True,
    db=SqliteDb(
        db_file=str(data_dir / "agent.db"),
        session_table="agent_sessions",
    ),
    add_history_to_context=True,
    markdown=True,
)
agent_os = AgentOS(agents=\[agent\])
app = agent_os.get_app()
if __name__ == "__main__":
    print("\n🚀 Starting service...")
    print("📍 http://localhost:7777")
    print("💡 Please upload documents to the knowledge base in the UI\n")
    agent_os.serve(app="knowledge_agent:app", port=7777, reload=False)
```

**(1) Running the Agent**

```
**python** **knowledge_agent.py**
```

![](https://assets.zilliz.com/1_df885706cf.png)

### 3. Connecting to the AgentOS Console

https://os.agno.com/

**(1) Create an Account and Sign In**

![](https://assets.zilliz.com/2_db0af51e58.png)

**(2) Connect Your Agent to AgentOS**

![](https://assets.zilliz.com/3_0a8c6f9436.png)

**(3) Configure the Exposed Port and Agent Name**

![](https://assets.zilliz.com/4_3844011799.png)

**(4) Add Documents and Index Them in Milvus**

![](https://assets.zilliz.com/5_776ea7ca11.png)

![](https://assets.zilliz.com/6_90b97c4660.png)

![](https://assets.zilliz.com/7_a98262d8c5.png)

![](https://assets.zilliz.com/8_58b7d77eea.png)

**(5) Test the Agent End to End**

![](https://assets.zilliz.com/_6e61038ba5.png)

In this setup, Milvus handles high-performance semantic retrieval. When the knowledge-base assistant receives a technical question, it invokes the `search_knowledge` tool to embed the query, retrieves the most relevant document chunks from Milvus, and uses those results as the basis for its response.

Milvus offers three deployment options, allowing you to choose an architecture that fits your operational requirements while keeping the application-level APIs consistent across all deployment modes.

The demo above shows the core retrieval and generation flow. To move this design into a production environment, however, several architectural aspects need to be discussed in more detail.

## How Retrieval Results Are Shared Across Agents

Agno's Team mode has a `share_member_interactions=True` option that allows later agents to inherit the full interaction history of earlier agents. In practice, this means that when the first agent retrieves information from Milvus, subsequent agents can reuse those results instead of running the same search again.

-   **The upside:** Retrieval costs are amortized across the team. One vector search supports multiple agents, reducing redundant queries.
    
-   **The downside:** Retrieval quality gets amplified. If the initial search returns incomplete or inaccurate results, that error propagates to every agent that depends on it.
    

This is why retrieval accuracy matters even more in multi-agent systems. A bad retrieval doesn't just degrade one agent's response—it affects the entire team.

  

Here's an example Team setup:

```
from agno.team import Team
analyst = Agent(
    name="Data Analyst",
    model=OpenAIChat(id="gpt-4o"),
    knowledge=knowledge_base,
    search_knowledge=True,
    instructions=\["Analyze data and extract key metrics"\]
)
writer = Agent(
    name="Report Writer",
    model=OpenAIChat(id="gpt-4o"),
    knowledge=knowledge_base,
    search_knowledge=True,
    instructions=\["Write reports based on the analysis results"\]
)
team = Team(
    agents=\[analyst, writer\],
    share_member_interactions=True,  *# Share knowledge retrieval results*
)
```

## Why Agno and Milvus Are Layered Separately

In this architecture, **Agno** sits at the conversation and orchestration layer. It is responsible for managing dialogue flow, coordinating agents, and maintaining conversational state, with session history persisted in a relational database. The system’s actual domain knowledge—such as product documentation and technical reports—is handled separately and stored as vector embeddings in **Milvus**. This clear division keeps conversational logic and knowledge storage fully decoupled.

Why this matters operationally:

-   **Independent scaling**: As Agno demand grows, add more Agno instances. As query volume grows, expand Milvus by adding query nodes. Each layer scales in isolation.
    
-   **Different hardware needs**: Agno is CPU- and memory-bound (LLM inference, workflow execution). Milvus is optimized for high-throughput vector retrieval (disk I/O, sometimes GPU acceleration). Separating them prevents resource contention.
    
-   **Cost optimization**: You can tune and allocate resources for each layer independently.
    

This layered approach gives you a more efficient, resilient, and production-ready architecture.

## What to Monitor When Using Agno with Milvus

Agno has built-in evaluation capabilities, but adding Milvus expands what you should watch. Based on our experience, focus on three areas:

-   **Retrieval quality**: Are the documents Milvus returns actually relevant to the query, or just superficially similar at the vector level?
    
-   **Answer faithfulness**: Is the final response grounded in the retrieved content, or is the LLM generating unsupported claims?
    
-   **End-to-end latency breakdown**: Don't just track total response time. Break it down by stage—embedding generation, vector search, context assembly, LLM inference—so you can identify where slowdowns occur.
    

**A practical example:** When your Milvus collection grows from 1 million to 10 million vectors, you might notice retrieval latency creeping up. That's usually a signal to tune index parameters (like `nlist` and `nprobe`) or consider moving from standalone to a distributed deployment.

## Conclusion

Building production-ready agent systems takes more than wiring together LLM calls and retrieval demos. You need clear architectural boundaries, infrastructure that scales independently, and observability to catch issues early.

In this post, I walked through how Agno and Milvus can work together: Agno for multi-agent orchestration, Milvus for scalable memory and semantic retrieval. By keeping these layers separate, you can move from prototype to production without rewriting core logic—and scale each component as needed.

If you're experimenting with similar setups, I'd be curious to hear what's working for you.

**Questions about Milvus?** Join our [Slack channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session.

How to Build Production-Ready Multi-Agent Systems with Agno and Milvus


What Is OpenClaw? Complete Guide to the Open-Source AI Agent


OpenClaw (Formerly Clawdbot & Moltbot) Explained: A Complete Guide to the Autonomous AI Agent



If you've been on tech Twitter, Hacker News, or Discord this week, you've seen it. A lobster emoji 🦞, screenshots of tasks being completed, and one bold claim: an AI that doesn't just *talk*—it actually *does*.

![](https://assets.zilliz.com/ST_1_567975a33f.png)

It got weirder over the weekend. Entrepreneur Matt Schlicht launched [Moltbook](https://moltbook.com)—a Reddit-style social network where only AI agents can post, and humans can only watch. Within days, over 1.5 million agents signed up. They formed communities, debated philosophy, complained about their human operators, and even founded their own religion called "Crustafarianism." Yes, really.
![](https://assets.zilliz.com/ST_2_b570b3e59b.png)

Welcome to the OpenClaw craze.

The hype is so real that Cloudflare's stock jumped 14% simply because developers use its infrastructure to run applications. Mac Mini sales reportedly spiked as people buy dedicated hardware for their new AI employee. And the GitHub repo? Over [150,000 stars](https://github.com/openclaw/openclaw) in just a few weeks.

So naturally, we had to show you how to set up your own OpenClaw instance—and connect it to Slack so you can boss around your AI assistant from your favorite messaging app.

## What Is OpenClaw?

[OpenClaw](https://openclaw.ai/) (formerly known as Clawdbot/Moltbot) is an open-source, autonomous AI agent that runs locally on user machines and performs real-world tasks via messaging apps such as WhatsApp, Telegram, and Discord. It automates digital workflows—such as managing emails, browsing the web, or scheduling meetings—by connecting to LLMs like Claude or ChatGPT. 

In short, it's like having a 24/7 digital assistant that can think, respond, and actually get stuff done.


## Setting Up OpenClaw as a Slack-Based AI Assistant

Imagine having a bot in your Slack workspace that can instantly answer questions about your product, help debug user issues, or point teammates to the right documentation—without anyone having to stop what they're doing. For us, that could mean faster support for the Milvus community: a bot that answers common questions ("How do I create a collection?"), helps troubleshoot errors, or summarizes release notes on demand. For your team, it might be onboarding new engineers, handling internal FAQs, or automating repetitive DevOps tasks. The use cases are wide open.

In this tutorial, we'll walk through the basics: installing OpenClaw on your machine and connecting it to Slack. Once that's done, you'll have a working AI assistant ready to customize for whatever you need.

### Prerequisites

-   A Mac or Linux machine
    
-   An [Anthropic API key](https://console.anthropic.com/) (or Claude Code CLI access)
    
-   A Slack workspace where you can install apps
    

That's it. Let's get started.

### Step 1: Install OpenClaw

Run the installer:

curl -fsSL https://molt.bot/install.sh | bash
![](https://assets.zilliz.com/ST_3_fc80684811.png)

  
When prompted, select **Yes** to continue.
![](https://assets.zilliz.com/ST_4_8004e87516.png)

  
Then, choose **QuickStart** mode.
![](https://assets.zilliz.com/ST_5_b5803c1d89.png)

  
### Step 2: Choose Your LLM

The installer will ask you to pick a model provider. We're using Anthropic with the Claude Code CLI for authentication.

1.  Select **Anthropic** as the provider
![](https://assets.zilliz.com/ST_6_a593124f6c.png)


2.  Complete the verification in your browser when prompted.
![](https://assets.zilliz.com/ST_7_410c1a39d3.png)


3.  Choose **anthropic/claude-opus-4-5-20251101** as your default model
![](https://assets.zilliz.com/ST_8_0c22bf5a16.png)

  
### Step 3: Set Up Slack

When asked to select a channel, choose **Slack.**
![](https://assets.zilliz.com/ST_9_cd4dfa5053.png)

  
Proceed to name your bot. We called ours "Clawdbot_Milvus."
![](https://assets.zilliz.com/ST_10_89c79ccd0d.png)

  
Now you'll need to create a Slack app and grab two tokens. Here's how:
![](https://assets.zilliz.com/ST_11_50df3aec5d.png)

**3.1 Create a Slack App**

Go to the [Slack API website](https://api.slack.com/apps?new_app=1) and create a new app from scratch.
![](https://assets.zilliz.com/ST_12_21987505d5.png)

  
Give it a name and select the workspace you want to use.
![](https://assets.zilliz.com/ST_13_7fce24b5c7.png)

**3.2 Set Bot Permissions**

In the sidebar, click **OAuth & Permissions**. Scroll down to **Bot Token Scopes** and add the permissions your bot needs.
![](https://assets.zilliz.com/ST_14_b08d66b55a.png)


**3.3 Enable Socket Mode**

Click **Socket Mode** in the sidebar and toggle it on.
![](https://assets.zilliz.com/ST_15_11545f95f8.png)


This will generate an **App-Level Token** (starts with `xapp-`). Copy it somewhere safe.
![](https://assets.zilliz.com/ST_16_c446eefd7d.png)


**3.4 Enable Event Subscriptions**

Go to **Event Subscriptions** and toggle it on.
![](https://assets.zilliz.com/ST_17_98387d6226.png)


Then choose which events your bot should subscribe to.
![](https://assets.zilliz.com/ST_18_b2a16d7786.png)

  
**3.5 Install the App**

Click **Install App** in the sidebar, then **Request to Install** (or install directly if you're a workspace admin).
![](https://assets.zilliz.com/ST_19_a5e76d0d33.png)

  
Once approved, you'll see your **Bot User OAuth Token** (starts with `xoxb-`). Copy this as well.
![](https://assets.zilliz.com/oauth_tokens_2e75e66f89.png)


### Step 4: Configure OpenClaw

Back in the OpenClaw CLI:

1.  Enter your **Bot User OAuth Token** (`xoxb-...`)
    
2.  Enter your **App-Level Token** (`xapp-...`)
![](https://assets.zilliz.com/ST_21_bd1629fb6a.png)


3.  Select which Slack channels the bot can access
![](https://assets.zilliz.com/ST_22_a1b682fa84.png)


4.  Skip skills configuration for now—you can always add them later
![](https://assets.zilliz.com/ST_23_cc4855ecfd.png)


5.  Select **Restart** to apply your changes
    

### Step 5: Try It Out

Head over to Slack and send your bot a message. If everything's set up correctly, OpenClaw will respond and be ready to run tasks on your machine.
![](https://assets.zilliz.com/ST_24_2864a88ce9.png)


### Tips

1.  Run `clawdbot dashboard` to manage settings through a web interface
![](https://assets.zilliz.com/ST_25_67b337b1d9.png)


2.  If something goes wrong, check the logs for error details
![](https://assets.zilliz.com/ST_26_a62b5669ee.png)


## A Word of Caution

OpenClaw is powerful—and that's exactly why you should be careful. "Actually does things" means it can execute real commands on your machine. That's the whole point, but it comes with risk.

**The good news:**

-   It's open source, so the code is auditable
    
-   It runs locally, so your data isn't on someone else's server
    
-   You control what permissions it has
    

**The not-so-good news:**

-   Prompt injection is a real risk—a malicious message could potentially trick the bot into running unintended commands
    
-   Scammers have already created fake OpenClaw repos and tokens, so be careful what you download
    

**Our advice:**

-   Don't run this on your primary machine. Use a VM, a spare laptop, or a dedicated server.
    
-   Don't grant more permissions than you need.
    
-   Don't use this in production yet. It's new. Treat it like the experiment it is.
    
-   Stick to official sources: [@openclaw](https://x.com/openclaw) on X and [OpenClaw](https://github.com/openclaw). 
    

Once you give an LLM the ability to execute commands, there's no such thing as 100% secure. That's not an OpenClaw problem—that's the nature of agentic AI. Just be smart about it.

## What's Next?

Congratulations! You now have a local AI assistant running on your own infrastructure, accessible through Slack. Your data stays yours, and you've got a tireless helper ready to automate the repetitive stuff.

From here, you can:

-   Install more [Skills](https://docs.molt.bot/skills) to expand what OpenClaw can do
    
-   Set up scheduled tasks so it works proactively
    
-   Connect other messaging platforms like Telegram or Discord
    
-   Explore the [Milvus](https://milvus.io/) ecosystem for AI search capabilities
    

**Have questions or want to share what you're building?**

-   Join the [Milvus Slack community](https://milvus.io/slack) to connect with other developers
    
-   Book our [Milvus Office Hours](https://milvus.io/office-hours) for live Q&A with the team
    

Happy hacking! 🦞

OpenClaw Tutorial: Connect to Slack for Local AI Assistant


Step-by-Step Guide to Setting Up OpenClaw (Previously Clawdbot/Moltbot) with Slack


Build Clawdbot-Style AI Agents with LangGraph & Milvus


Why Clawdbot Went Viral — And How to Build Production-Ready Long-Running Agents with LangGraph and Milvus


How Claude Code Manages Local Storage for AI Agents


Why Claude Code Feels So Stable: A Developer’s Deep Dive into Its Local Storage Design



[Claude Cowork](https://support.claude.com/en/articles/13345190-getting-started-with-cowork) is a new agent feature in the Claude Desktop app. From a developer’s point of view, it’s basically an automated task runner wrapped around the model: it can read, modify, and generate local files, and it can plan multi-step tasks without you having to manually prompt for each step. Think of it as the same loop behind Claude Code, but exposed to the desktop instead of the terminal. 

Cowork’s key capability is its ability to run for extended periods without losing state. It doesn’t hit the usual conversation timeout or context reset. It can keep working, track intermediate results, and reuse previous information across sessions. That gives the impression of “long-term memory,” even though the underlying mechanics are more like persistent task state + contextual carryover. Either way, the experience is different from the traditional chat model, where everything resets unless you build your own memory layer.

This brings up two practical questions for developers:

1.  **If the model can already remember past information, where does RAG or agentic RAG still fit in? Will RAG be replaced?** 
    
2.  **If we want a local, Cowork-style agent, how do we implement long-term memory ourselves?**
    

The rest of this article addresses these questions in detail and explains how vector databases fit into this new “model memory” landscape.

## Claude Cowork vs. RAG: What’s the Difference? 

As I mentioned previously, Claude Cowork is an agent mode inside Claude Desktop that can read and write local files, break tasks into smaller steps, and keep working without losing state. It maintains its own working context, so multi-hour tasks don’t reset like a normal chat session.

**RAG** (Retrieval-Augmented Generation) solves a different problem: giving a model access to external knowledge. You index your data into a vector database, retrieve relevant chunks for each query, and feed them into the model. It’s widely used because it provides LLM applications with a form of “long-term memory” for documents, logs, product data, and more.

If both systems help a model “remember,” what’s the actual difference?

### How Cowork Handles Memory

Cowork’s memory is read-write. The agent decides which information from the current task or conversation is relevant, stores it as memory entries, and retrieves it later as the task progresses. This allows Cowork to maintain continuity across long-running workflows — especially ones that produce new intermediate state as they progress.

### How RAG and Agentic RAG Handle Memory

Standard RAG is query-driven retrieval: the user asks something, the system fetches relevant documents, and the model uses them to answer. The retrieval corpus stays stable and versioned, and developers control exactly what enters it.

Modern agentic RAG extends this pattern. The model can decide when to retrieve information, what to retrieve, and how to use it during the planning or execution of a workflow. These systems can run long tasks and call tools, similar to Cowork. But even with agentic RAG, the retrieval layer remains knowledge-oriented rather than state-oriented. The agent retrieves authoritative facts; it doesn’t write its evolving task state back into the corpus.

Another way to look at it:

-   **Cowork’s memory is task-driven:** the agent writes and reads its own evolving state.
    
-   **RAG is knowledge-driven:** the system retrieves established information that the model should rely on.
    

## Reverse-Engineering Claude Cowork: How It Builds Long-Running Agent Memory

Cowork gets a lot of hype because it handles multi-step tasks without constantly forgetting what it was doing. From a developer's perspective, I am wondering **how it keeps state across such long sessions?** Anthropic hasn’t published the internals, but based on earlier dev experiments with Claude’s memory module, we can piece together a decent mental model.

Claude seems to rely on a hybrid setup: **a persistent long-term memory layer plus on-demand retrieval tools.** Instead of stuffing the full conversation into every request, Claude selectively pulls in past context only when it decides it’s relevant. This lets the model keep accuracy high without blowing through tokens every turn.

If you break down the request structure, it roughly looks like this:

```
[0] Static system instructions
[1] User memory (long-term)
[2] Retrieved / pruned conversation history
[3] Current user message
```

The interesting behavior isn’t the structure itself — it’s how the model decides what to update and when to run retrieval.

### User Memory: The Persistent Layer

Claude keeps a long-term memory store that updates over time. And unlike ChatGPT’s more predictable memory system, Claude’s feels a bit more “alive.” It stores memories in XML-ish blocks and updates them in two ways: 

-   **Implicit updates:** Sometimes the model just decides something is a stable preference or fact and quietly writes it to memory. These updates aren’t instantaneous; they show up after a few turns, and older memories can fade out if the related conversation disappears.
    
-   **Explicit updates:** Users can directly modify memory with the `memory_user_edits` tool (“remember X,” “forget Y”). These writes are immediate and behave more like a CRUD operation.
    

Claude is running background heuristics to decide what’s worth persisting, and it’s not waiting for explicit instructions.

### Conversation Retrieval: The On-Demand Part

Claude does *not* keep a rolling summary like many LLM systems. Instead, it has a toolbox of retrieval functions it can call whenever it thinks it’s missing context. These retrieval calls don’t happen every turn — the model triggers them based on its own internal judgment. 

The standout is `conversation_search`. When the user says something vague like “that project from last month,” Claude often fires this tool to dig up relevant turns. What’s notable is that it still works when the phrasing is ambiguous or in a different language. That pretty clearly implies:

-   Some kind of semantic matching (embeddings)
    
-   Probably combined with normalization or lightweight translation
    
-   Keyword search layered in for precision
    

Basically, this looks a lot like a miniature RAG system bundled inside the model’s toolset.

### How Claude’s Retrieval Behavior Differs From Basic History Buffers

From testing and logs, a few patterns stand out:

-   **Retrieval isn’t automatic.** The model chooses when to call it. If it thinks it already has enough context, it won’t even bother.
    
-   **Retrieved chunks include** *both* **user and assistant messages.** That’s useful — it keeps more nuance than user-only summaries.
    
-   **Token usage stays sane.** Because history isn’t injected every turn, long sessions don’t balloon unpredictably.
    

Overall, it feels like a retrieval-augmented LLM, except the retrieval happens as part of the model’s own reasoning loop.

This architecture is clever, but not free:

-   Retrieval adds latency and more “moving parts” (indexing, ranking, re-ranking).
    
-   The model occasionally misjudges whether it needs context, which means you see the classic “LLM forgetfulness” even though the data *was* available.
    
-   Debugging becomes trickier because model behavior depends on invisible tool triggers, not just prompt input.
    

### Claude Cowork vs Claude Codex in handling long-term memory 

In contrast to Claude’s retrieval-heavy setup, ChatGPT handles memory in a much more structured and predictable way. Instead of doing semantic lookups or treating old conversations like a mini vector store, ChatGPT injects memory directly into each session through the following layered components: 

-   User memory
    
-   Session metadata
    
-   Current session messages
    

**User Memory**

User Memory is the main long-term storage layer—the part that persists across sessions and can be edited by the user. It stores pretty standard things: name, background, ongoing projects, learning preferences, that kind of stuff. Every new conversation gets this block injected at the start, so the model always starts with a consistent view of the user.

ChatGPT updates this layer in two ways:

-   **Explicit updates:** Users can tell the model to “remember this” or “forget that,” and the memory changes immediately. This is basically a CRUD API that the model exposes through natural language.
    
-   **Implicit updates:** If the model spots information that fits OpenAI’s rules for long-term memory—like a job title or a preference—and the user hasn’t disabled memory, it will quietly add it on its own.
    

From a developer angle, this layer is simple, deterministic, and easy to reason about. No embedding lookups, no heuristics about what to fetch.

**Session Metadata**

Session Metadata sits at the opposite end of the spectrum. It’s short-lived, non-persistent, and only injected once at the start of a session. Think of it as environment variables for the conversation. This includes things like:

-   what device you’re on
    
-   account/subscription state
    
-   rough usage patterns (active days, model distribution, average conversation length)
    

This metadata helps the model shape responses for the current environment—e.g., writing shorter answers on mobile—without polluting long-term memory.

**Current Session Messages**

This is the standard sliding-window history: all messages in the current conversation until the token limit is reached. When the window gets too large, older turns drop off automatically.

Crucially, this eviction **does not** touch User Memory or cross-session summaries. Only the local conversation history shrinks.

The biggest divergence from Claude appears in how ChatGPT handles “recent but not current” conversations. Claude will call a search tool to retrieve past context if it thinks it’s relevant. ChatGPT does not do that.

Instead, ChatGPT keeps a very lightweight **cross-session summary** that gets injected into every conversation. A few key details about this layer:

-   It summarizes **only user messages**, not assistant messages.
    
-   It stores a very small set of items—roughly 15—just enough to capture stable themes or interests.
    
-   It does **no embedding computation, no similarity ranking, and no retrieval calls**. It’s basically pre-chewed context, not dynamic lookup.
    

From an engineering perspective, this approach trades flexibility for predictability. There’s no chance of a weird retrieval failure, and inference latency stays stable because nothing is being fetched on the fly. The downside is that ChatGPT won’t pull in some random message from six months ago unless it made it into the summary layer.

## Challenges to Making Agent Memory Writable

When an agent moves from **read-only memory** (typical RAG) to **writable memory**—where it can log user actions, decisions, and preferences—the complexity jumps quickly. You’re no longer just retrieving documents; you’re maintaining a growing state on which the model depends. 

A writable memory system has to solve three real problems:

1.  **What to remember:** The agent needs rules for deciding which events, preferences, or observations are worth keeping. Without this, memory either explodes in size or fills with noise.
    
2.  **How to store and tier memory:** Not all memory is equal. Recent items, long-term facts, and ephemeral notes all need different storage layers, retention policies, and indexing strategies.
    
3.  **How to write fast without breaking retrieval:** Memory must be written continuously, but frequent updates can degrade index quality or slow queries if the system isn’t designed for high-throughput inserts.
    

### Challenge 1: What Is Worth Remembering?

Not everything a user does should end up in long-term memory. If someone creates a temp file and deletes it five minutes later, recording that forever doesn’t help anyone. This is the core difficulty: **how does the system decide what actually matters?** 

**(1) Common ways to judge importance**

Teams usually rely on a mix of heuristics:

-   **Time-based**: recent actions matter more than old ones
    
-   **Frequency-based**: files or actions accessed repeatedly are more important
    
-   **Type-based**: some objects are inherently more important (for example, project config files vs. cache files)
    

**(2) When rules conflict**

These signals often conflict. A file created last week but edited heavily today—should age or activity win? There’s no single “correct” answer, which is why importance scoring tends to get messy fast.

**(3) How vector databases help** 

Vector databases give you mechanisms to enforce importance rules without manual cleanup:

-   **TTL:** Milvus can automatically remove data after a set time
    
-   **Decay:** older vectors can be down-weighted so they naturally fade from retrieval
    

### Challenge 2: Memory Tiering in Practice

As agents run longer, memory piles up. Keeping everything in fast storage isn’t sustainable, so the system needs a way to split memory into **hot** (frequently accessed) and **cold** (rarely accessed) tiers.

**(1) Deciding When Memory Becomes Cold**

In this model, *hot memory* refers to data kept in RAM for low-latency access, while *cold memory* refers to data moved to disk or object storage to reduce cost. 

Deciding when memory becomes cold can be handled in different ways. Some systems use lightweight models to estimate the semantic importance of an action or file based on its meaning and recent usage. Others rely on simple, rule-based logic, such as moving memory that has not been accessed for 30 days or has not appeared in retrieval results for a week. Users may also explicitly mark certain files or actions as important, ensuring they always remain hot.

**(2) Where Hot and Cold Memory Are Stored**

Once classified, hot and cold memories are stored differently. Hot memory stays in RAM and is used for frequently accessed content, such as active task context or recent user actions. Cold memory is moved to disk or object storage systems like S3, where access is slower but storage costs are much lower. This trade-off works well because cold memory is rarely needed and is typically accessed only for long-term reference.

**(3) How Vector Databases Help**

**Milvus and Zilliz Cloud** support this pattern by enabling hot–cold tiered storage while maintaining a single query interface, so frequently accessed vectors stay in memory and older data moves to lower-cost storage automatically.

### Challenge 3: How Fast Should Memory Be Written?

Traditional RAG systems usually write data in batches. Indexes are rebuilt offline—often overnight—and only become searchable later. This approach works for static knowledge bases, but it does not fit agent memory.

**(1) Why Agent Memory Needs Real-Time Writes**

Agent memory must capture user actions as they happen. If an action is not recorded immediately, the next conversation turn may lack critical context. For this reason, writable memory systems require real-time writes rather than delayed, offline updates.

**(2) The Tension Between Write Speed and Retrieval Quality**

Real-time memory demands very low write latency. At the same time, high-quality retrieval depends on well-built indexes, and index construction takes time. Rebuilding an index for every write is too expensive, but delaying indexing means newly written data remains temporarily invisible to retrieval. This trade-off sits at the center of writable memory design.

**(3) How Vector Databases Help**

Vector databases address this problem by decoupling writing from indexing. A common solution is to stream writes and perform incremental index builds. Using **Milvus** as an example, new data is first written to an in-memory buffer, allowing the system to handle high-frequency writes efficiently. Even before a full index is built, buffered data can be queried within seconds through dynamic merging or approximate search.

When the buffer reaches a predefined threshold, the system builds indexes in batches and persists them. This improves long-term retrieval performance without blocking real-time writes. By separating fast ingestion from slower index construction, Milvus achieves a practical balance between write speed and search quality that works well for agent memory.

## Conclusion 

Cowork gives us a glimpse of a new class of agents—persistent, stateful, and able to carry context across long timelines. But it also makes something else clear: long-term memory is only half of the picture. To build production-ready agents that are both autonomous and reliable, we still need structured retrieval over large, evolving knowledge bases.

RAG handles the world’s facts; writable memory handles the agent’s internal state. And vector databases sit at the intersection, providing indexing, hybrid search, and scalable storage that enable both layers to work together.

As long-running agents continue to mature, their architectures will likely converge on this hybrid design. Cowork is a strong signal of where things are heading—not toward a world without RAG, but toward agents with richer memory stacks powered by vector databases underneath.

If you want to explore these ideas or get help with your own setup, **join our** [Slack Channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) to chat with Milvus engineers. And for more hands-on guidance, you can always **book a** [**Milvus Office Hours**](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) **session.**


RAG vs Long-Running Agents: Is RAG Obsolete? 


Is RAG Becoming Outdated Now That Long-Running Agents Like Claude Cowork Are Emerging?



Tool usage is a big part of making an agent work. The agent needs to choose the right tool, decide when to call it, and format the inputs correctly. On paper that sounds straightforward, but once you start building real systems, you find a lot of edge cases and failure modes.

Many teams use MCP-style tool definitions to organize this, but MCP has some rough edges. The model has to reason over all tools at once, and there isn’t much structure to guide its decisions. On top of that, every tool definition has to live in the context window. Some of these are large — the GitHub MCP is around 26k tokens — which eats context before the agent even starts doing real work.

Anthropic introduced [**Skills**](https://github.com/anthropics/skills?tab=readme-ov-file) to improve this situation. Skills are smaller, more focused, and easier to load on demand. Instead of dumping everything into context, you package domain logic, workflows, or scripts into compact units that the agent can pull in only when needed.

In this post, I’ll go over how Anthropic Skills work and then walk through building a simple Skill in Claude Code that turns natural language into a [Milvus](https://milvus.io/)-backed knowledge base — a quick setup for RAG without extra wiring.

## What Are Anthropic Skills?

[Anthropic Skills](https://milvus.io/blog/is-mcp-already-outdated-the-real-reason-anthropic-shipped-skills-and-how-to-pair-them-with-milvus.md) (or Agent Skills) are just folders that bundle the instructions, scripts, and reference files an agent needs to handle a specific task. Think of them as small, self-contained capability packs. A Skill might define how to generate a report, run an analysis, or follow a particular workflow or set of rules.

The key idea is that Skills are modular and can be loaded on demand. Instead of stuffing huge tool definitions into the context window, the agent pulls in only the Skill it needs. This keeps context usage low while giving the model clear guidance on what tools exist, when to call them, and how to execute each step.

The format is intentionally simple, and because of that, it’s already supported or easily adapted across a bunch of developer tools — Claude Code, Cursor, VS Code extensions, GitHub integrations, Codex-style setups, and so on.

A Skill follows a consistent folder structure:

```
skill-name/

├── SKILL.md       # Required: Skill instructions and metadata

├── scripts/         # Optional: helper scripts

├── templates/       # Optional: document templates

└── resources/       # Optional: reference materials
```

**1.** `SKILL.md` **(Core File)**

This is the execution guide for the agent—the document that tells the agent exactly how the task should be carried out. It defines the Skill’s metadata (such as name, description, and trigger keywords), the execution flow, and default settings. In this file, you should clearly describe:

- **When the Skill should run:** For example, trigger the Skill when the user input includes a phrase like “process CSV files with Python.”
    
- **How the task should be performed:** Lay out the execution steps in order, such as: interpret the user’s request → call preprocessing scripts from the `scripts/` directory → generate the required code → format the output using templates from `templates/`.
    
- **Rules and constraints:** Specify details such as coding conventions, output formats, and how errors should be handled.
    

**2.** `scripts/` **(Execution Scripts)**

This directory contains prewritten scripts in languages such as Python, Shell, or Node.js. The agent can call these scripts directly, instead of generating the same code repeatedly at runtime. Typical examples include `create_collection.py` and `check_env.py`.

**3.** `templates/` **(Document Templates)**

Reusable template files that the agent can use to generate customized content. Common examples include report templates or configuration templates.

**4.** `resources/` **(Reference Materials)**

Reference documents the agent can consult during execution, such as API documentation, technical specifications, or best-practice guides.

Overall, this structure mirrors how work is handed off to a new teammate: `SKILL.md` explains the job, `scripts/` provide ready-to-use tools, `templates/` define standard formats, and `resources/` supply background information. With all of this in place, the agent can execute the task reliably and with minimal guesswork.

## Hands-on Tutorial: Creating a Custom Skill for a Milvus-Powered RAG System

In this section, we’ll walk through building a custom Skill that can set up a Milvus collection and assemble a full RAG pipeline from plain natural-language instructions. The goal is to skip all the usual setup work — no manual schema design, no index configuration, no boilerplate code. You tell the agent what you want, and the Skill handles the Milvus pieces for you.

### Design Overview

![](https://assets.zilliz.com/design_overview_d4c886291b.PNG)

### Prerequisites

  

| Component | Requirement |
| --- | --- |
| CLI | `claude-code` |
| Models | GLM 4.7, OpenAI |
| Container | Docker |
| Milvus | 2.6.8 |
| Model Configuration Platform | CC-Switch |
| Package Manager | npm |
| Development Language | Python |

  

### Step 1: Environment Setup

**Install** `claude-code`

```
npm install -g @anthropic-ai/claude-code
```

**Install CC-Switch**

**Note:** CC-Switch is a model-switching tool that makes it easy to switch between different model APIs when running AI models locally.

Project repository: [https://github.com/farion1231/cc-switch](https://github.com/farion1231/cc-switch)

**Select Claude and Add an API Key**

![](https://assets.zilliz.com/1_0cdfab2e54.PNG)

![](https://assets.zilliz.com/2_615ee13649.PNG)

**Check the Current Status**

![](https://assets.zilliz.com/3_f1c13da1fe.PNG)

**Deploy and Start Milvus-Standalone**

```
# Download docker-compose.yml

wget https://github.com/milvus-io/milvus/releases/download/v2.6.8/milvus-standalone-docker-compose.yml -O docker-compose.yml

  

# Start Milvus (check port mapping: 19530:19530)

docker-compose up -d

  

# Verify that the services are running

docker ps | grep milvus

# You should see three containers: milvus-standalone, milvus-etcd, milvus-minio
```

![](https://assets.zilliz.com/code1_9c6a1a7f93.PNG)

**Configure the OpenAI API Key**

```
# Add this to ~/.bashrc or ~/.zshrc

OPENAI_API_KEY=your_openai_api_key_here
```

### Step 2: Create the Custom Skill for Milvus

**Create the Directory Structure**

```
cd ~/.claude/skills/

mkdir -p milvus-skills/example milvus-skills/scripts
```

**Initialize** `SKILL.md`

**Note:** SKILL.md serves as the agent’s execution guide. It defines what the Skill does and how it should be triggered.

```
name: milvus-collection-builder

description: Create Milvus collections using natural language, supporting both RAG and text search scenarios
```

**Write the Core Scripts**

| Script Type | File Name | Purpose |
| --- | --- | --- |
| Environment check | `check_env.py` | Checks the Python version, required dependencies, and the Milvus connection |
| Intent parsing | `intent_parser.py` | Converts requests like “build a RAG database” into a structured intent such as `scene=rag` |
| Collection creation | `milvus_builder.py` | The core builder that generates the collection schema and index configuration |
| Data ingestion | `insert_milvus_data.py` | Loads documents, chunks them, generates embeddings, and writes data into Milvus |
| Example 1 | `basic_text_search.py` | Demonstrates how to create a document search system |
| Example 2 | `rag_knowledge_base.py` | Demonstrates how to build a complete RAG knowledge base |

These scripts show how to turn a Milvus-focused Skill into something practical: a working document search system and an intelligent Q&A (RAG) setup.

### Step 3: Enable the Skill and Run a Test

**Describe the Request in Natural Language**

```
"I want to build an RAG system."
```

![](https://assets.zilliz.com/test1_64fd549573.PNG)

**RAG System Created**

![](https://assets.zilliz.com/test2_80656d59b1.png)

**Insert Sample Data**

![](https://assets.zilliz.com/test3_392753eb73.png)

**Run a Query**

![](https://assets.zilliz.com/test4_75e23c6a3a.png)

## Conclusion

In this tutorial, we walked through building a Milvus-powered RAG system using a custom Skill. The goal wasn’t just to show another way to call Milvus—it was to show how Skills can turn what is normally a multi-step, configuration-heavy setup into something you can reuse and iterate on. Instead of manually defining schemas, tuning indexes, or stitching together workflow code, the Skill handles most of the boilerplate so you can focus on the parts of RAG that actually matter.

This is only the start. A full RAG pipeline has plenty of moving pieces: preprocessing, chunking, hybrid search settings, reranking, evaluation, and more. All of these can be packaged as separate Skills and composed depending on your use case. If your team has internal standards for vector dimensions, index params, prompt templates, or retrieval logic, Skills are a clean way to encode that knowledge and make it repeatable.

For new developers, this lowers the entry barrier—no need to learn every detail of Milvus before getting something running. For experienced teams, it cuts down on repeated setup and helps keep projects consistent across environments. Skills won’t replace thoughtful system design, but they remove a lot of unnecessary friction.

👉 The full implementation is available in the [open-source repository](https://github.com/yinmin2020/open-milvus-skills), and you can explore more community-built examples in the [Skill marketplace](https://skillsmp.com/).

## Stay tuned!

We’re also working on introducing official Milvus and Zilliz Cloud Skills that cover common RAG patterns and production best practices. If you have ideas or specific workflows you want supported, join our [Slack Channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) and chat with our engineers. And if you want guidance for your own setup, you can always book a [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session.

Create a Custom Anthropic Skill for Milvus to Quickly Spin Up RAG


How Anthropic Skills Change Agent Tooling — and How to Build a Custom Skill for Milvus to Quickly Spin Up RAG



As AI development accelerates, more teams are discovering that building an AI assistant doesn’t necessarily require a software engineering background. The people who need assistants the most—product teams, operations, support, researchers—often know exactly what the agent should do, but not how to implement it in code. Traditional “no-code” tools tried to bridge that gap with drag-and-drop canvases, yet they collapse the moment you need real agent behavior: multi-step reasoning, tool use, or persistent memory. 

The newly released [**LangSmith Agent Builder**](https://www.langchain.com/langsmith/agent-builder) takes a different approach. Instead of designing workflows, you describe the agent’s goals and available tools in plain language, and the runtime handles the decision-making. No flowcharts, no scripting—just clear intent.

But intent alone doesn’t produce an intelligent assistant. **Memory** does. This is where [**Milvus**](https://milvus.io/), the widely adopted open-source vector database, provides the foundation. By storing documents and conversation history as embeddings, Milvus allows your agent to recall context, retrieve relevant information, and respond accurately at scale. 

This guide walks through how to build a production-ready, memory-enabled AI assistant using **LangSmith Agent Builder + Milvus**, all without writing a single line of code.

## What is LangSmith Agent Builder and How It Works? 

Just as its name reveals, [LangSmith Agent Builder](https://www.google.com/search?q=LangSmith+Agent+Builder&oq=what+is+LangSmith+Agent+Builder&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQABiABDIHCAIQABiABDIHCAMQABiABDIHCAQQABiABDIHCAUQABiABDIHCAYQABiABDIHCAcQABiABDIGCAgQABgDMggICRAAGBYYHtIBCTI1OTJqMGoxNagCCLACAfEF2Mylr_IuXLk&sourceid=chrome&ie=UTF-8&ved=2ahUKEwjV1LfvxZ6SAxVFsFYBHYzTJAsQgK4QegQIARAB) is a no-code tool from LangChain that lets you build, deploy, and manage AI agents using plain language. Instead of writing logic or designing visual flows, you explain what the agent should do, what tools it can use, and how it should behave. The system then handles the hard parts—generating prompts, selecting tools, wiring components together, and enabling memory. 

![](https://assets.zilliz.com/_57c5cee35b.png)

Unlike traditional no-code or workflow tools, Agent Builder doesn’t have drag-and-drop canvas and no node library. You interact with it the same way you would with ChatGPT. Describe what you want to build, answer a few clarifying questions, and the Builder produces a fully functioning agent based on your intent.  

Behind the scenes, that agent is constructed from four core building blocks.

![](https://assets.zilliz.com/640_05b90b1f3d.webp)

-   **Prompt:** The prompt is the agent’s brain, defining its goals, constraints, and decision logic. LangSmith Agent Builder uses meta-prompting to build this automatically: you describe what you want, it asks clarifying questions, and your answers are synthesized into a detailed, production-ready system prompt. Instead of hand-writing logic, you simply express intent.
-   **Tools:** Tools let the agent take action—sending emails, posting to Slack, creating calendar events, searching data, or calling APIs. Agent Builder integrates these tools through the Model Context Protocol (MCP), which provides a secure, extensible way to expose capabilities. Users can rely on built-in integrations or add custom MCP servers, including Milvus [MCP server](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)s for vector search and long-term memory.
-   **Triggers:** Triggers define when an agent runs. In addition to manual execution, you can attach agents to schedules or external events so they automatically respond to messages, emails, or webhook activity. When a trigger fires, Agent Builder starts a new execution thread and runs the agent’s logic, enabling continuous, event-driven behavior.
-   **Subagents:** Subagents break complex tasks into smaller, specialized units. A primary agent can delegate work to subagents—each with its own prompt and toolset—so tasks like data retrieval, summarization, or formatting are handled by dedicated helpers. This avoids a single overloaded prompt and creates a more modular, scalable agent architecture.

## How Does an Agent Remember Your Preferences?

What makes Agent Builder unique is how it treats *memory*. Instead of stuffing preferences into chat history, the agent can update its own behavior rules while running. If you say, “From now on, end every Slack message with a poem,” the agent doesn’t treat that as a one-off request—it stores it as a persistent preference that applies in future runs.

Under the hood, the agent keeps an internal memory file—essentially its evolving system prompt. Each time it starts, it reads this file to decide how to behave. When you give corrections or constraints, the agent edits the file by adding structured rules like “Always close the briefing with a short uplifting poem.” This approach is far more stable than relying on conversation history because the agent actively rewrites its operating instructions rather than burying your preferences inside a transcript.

This design comes from DeepAgents’ FilesystemMiddleware but is fully abstracted in Agent Builder. You never touch files directly: you express updates in natural language, and the system handles the edits behind the scenes. If you need more control, you can plug in a custom MCP server or drop to the DeepAgents layer for advanced memory customization.

## Hands-on Demo: Building a Milvus Assistant in 10 Minutes using Agent Builder 

Now that we’ve covered the design philosophy behind Agent Builder, let’s walk through the full build process with a hands-on example. Our goal is to create an intelligent assistant that can answer Milvus-related technical questions, search the official documentation, and remember user preferences over time. 

### Step 1. Sign In to the LangChain Website

![](https://assets.zilliz.com/640_1_b3c461d39b.webp)

### Step 2. Set Up Your Anthropic API Key

**Note:** Anthropic is supported by default. You can also use a custom model, as long as its type is included in the list officially supported by LangChain.

![](https://assets.zilliz.com/640_2_c04400695e.webp)

**1. Add an API Key** 

![](https://assets.zilliz.com/640_3_11db4b3824.webp)

**2. Enter and Save the API Key**

![](https://assets.zilliz.com/640_4_abfc27d796.webp)

### Step 3. Create a New Agent

**Note:** Click **Learn More** to view the usage documentation.

![](https://assets.zilliz.com/640_5_e90bf254f2.webp)

![](https://assets.zilliz.com/640_6_7c839d96f3.webp)

1.  **Configure a Custom Model (Optional)**

![](https://assets.zilliz.com/640_7_0dfd5ff561.webp)

**(1) Enter Parameters and Save**

![](https://assets.zilliz.com/640_8_85f9e3008f.webp)

![](https://assets.zilliz.com/640_9_0d5d0c062c.webp)

### Step 4. Describe Your Requirements to Create the Agent

**Note:** Create the agent using a natural language description.

```

I need a Milvus technical consultant to help me answer technical questions about vector databases. 

Search the official documentation and remember my preference for the index type I use. 

```

![](https://assets.zilliz.com/640_10_0e033a5200.webp)

1.  **The System Asks Follow-Up Questions to Refine Requirements**

Question 1: Select the Milvus index types you want the agent to remember

![](https://assets.zilliz.com/640_11_050ac891f0.webp)
  
Question 2: Choose how the agent should handle technical questions
![](https://assets.zilliz.com/640_12_d1d6d4f2ed.webp)
  
Question 3: Specify whether the agent should focus on guidance for a specific Milvus version
![](https://assets.zilliz.com/640_13_5d60df75e9.webp)

### Step 5. Review and Confirm the Generated Agent

**Note:** The system automatically generates the agent configuration.

![](https://assets.zilliz.com/640_14_8a596ae853.webp)

Before creating the agent, you can review its metadata, tools, and prompts. Once everything looks correct, click **Create** to proceed.

![](https://assets.zilliz.com/640_15_5c0b27aca7.webp)

![](https://assets.zilliz.com/640_16_998921b071.webp)

### Step 6. Explore the Interface and Feature Areas

After the agent is created, you’ll see three functional areas in the lower-left corner of the interface:

**(1) Triggers**

Triggers define when the agent should run, either in response to external events or on a schedule:

-   **Slack:** Activate the agent when a message arrives in a specific channel
-   **Gmail:** Trigger the agent when a new email is received
-   **Cron:** Run the agent on a scheduled interval

![](https://assets.zilliz.com/640_17_b77c603413.webp)

**(2) Toolbox**

This is the set of tools the agent can call. In the example shown, the three tools are generated automatically during creation, and you can add more by clicking **Add tool**.
![](https://assets.zilliz.com/640_18_94637d4548.webp)
 
**If your agent needs vector search capabilities—such as semantic search across large volumes of technical documentation—you can deploy Milvus’s MCP Server** and add it here using the **MCP** button. Make sure the MCP server is running **at a reachable network endpoint**; otherwise, Agent Builder won’t be able to invoke it.

![](https://assets.zilliz.com/640_19_94fe99a3b8.webp)
![](https://assets.zilliz.com/640_20_f887a8fbda.webp)

**(3) Sub-agents**

Create independent agent modules dedicated to specific subtasks, enabling a modular system design.

### Step 7. Test the Agent

Click **Test** in the top-right corner to enter testing mode. Below is a sample of the test results.

![](https://assets.zilliz.com/640_22_527619519b.webp)

![](https://assets.zilliz.com/640_23_639d40c40d.webp)  

![](https://assets.zilliz.com/640_24_42a71d2592.webp)

![](https://assets.zilliz.com/640_25_8ab35e15f8.webp)

## Agent Builder vs. DeepAgents: Which One Should You Choose?

LangChain offers multiple agent frameworks, and the right choice depends on how much control you need. [DeepAgents](https://www.google.com/search?q=DeepAgents&newwindow=1&sca_esv=0e7ec9ce2aa7d5b4&sxsrf=ANbL-n5pe1KqjmJVjQCqmc3jneYhmGGOUg%3A1769066335766&ei=X89xab21Lp3a1e8Ppam06Ag&ved=2ahUKEwio15nYzZ6SAxU_mq8BHcf3BqUQgK4QegQIARAB&uact=5&oq=what+is+DeepAgents&gs_lp=Egxnd3Mtd2l6LXNlcnAiEndoYXQgaXMgRGVlcEFnZW50czIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzIKEAAYsAMY1gQYRzINEAAYgAQYsAMYQxiKBTINEAAYgAQYsAMYQxiKBUi8BlAYWL8FcAF4AZABAJgBqgKgAbAKqgEFMC4yLjS4AQPIAQD4AQGYAgGgAgyYAwCIBgGQBgqSBwExoAehHrIHALgHAMIHAzMtMcgHCYAIAA&sclient=gws-wiz-serp) is agent building tool. It is used to build autonomous, long-running AI agents that handle complex, multi-step tasks. Built on LangGraph, it supports advanced planning, file-based context management, and subagent orchestration—making it ideal for long-horizon or production-grade projects.

So how does that compare to **Agent Builder**, and when should you use each?

**Agent Builder** focuses on simplicity and speed. It abstracts away most implementation details, letting you describe your agent in natural language, configure tools, and run it immediately. Memory, tool use, and human-in-the-loop workflows are handled for you. This makes Agent Builder perfect for rapid prototyping, internal tools, and early-stage validation where ease of use matters more than granular control.

**DeepAgents**, by contrast, is designed for scenarios where you need full control over memory, execution, and infrastructure. You can customize middleware, integrate any Python tool, modify the storage backend (including persisting memory in [Milvus](https://milvus.io/blog)), and explicitly manage the agent’s state graph. The trade-off is engineering effort—you write code, manage dependencies, and handle failure modes yourself—but you get a fully customizable agent stack.

Importantly, **Agent Builder and DeepAgents are not separate ecosystems—they form a single continuum**. Agent Builder is built on top of DeepAgents. That means you can start with a quick prototype in Agent Builder, then drop into DeepAgents when you need more flexibility, without rewriting everything from scratch. The reverse also works: patterns built in DeepAgents can be packaged as Agent Builder templates so non-technical users can reuse them.

## Conclusion

Thanks to the development of AI, building AI agents no longer requires complex workflows or heavy engineering. With LangSmith Agent Builder, you can create stateful, long-running assistants using natural language alone. You focus on describing what the agent should do, while the system handles planning, tool execution, and ongoing memory updates.

Paired with [Milvus](https://milvus.io/blog), these agents gain reliable, persistent memory for semantic search, preference tracking, and long-term context across sessions. Whether you’re validating an idea or deploying a scalable system, LangSmith Agent Builder and Milvus provide a simple, flexible foundation for agents that don’t just respond—they remember and improve over time.

Have questions or want a deeper walkthrough? Join our [Slack channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session for personalized guidance.

Build AI Agents in 10 Minutes with LangSmith Agent Builder & Milvus


Building AI Agents in 10 Minutes Using Natural Language with LangSmith Agent Builder + Milvus



[ScaNN](https://github.com/google-research/google-research/tree/master/scann) is Google’s answer to a familiar challenge in large-scale vector search: **how to increase query throughput and reduce memory usage without taking an unacceptable hit to result quality.** Conceptually, ScaNN starts from the classic IVF+PQ recipe—coarse clustering plus aggressive product quantization—but layers on two important innovations that meaningfully shift the performance frontier:

- **A score-aware quantization objective** that better preserves the relative ordering of true neighbors, improving ranking quality even under heavy compression.

- **FastScan** is a SIMD-optimized 4-bit PQ lookup path that reduces the traditional memory-load bottleneck by keeping more work inside CPU registers.

In practice, it is a strong choice when you are okay with trading some recall for high QPS and a much smaller memory footprint (often compressing vectors to ~1/16 of the original size), such as in recall-insensitive recommendation workloads.

In this post, we’ll revisit IVFPQ as the baseline, explore the key optimizations ScaNN introduces on top of it, and wrap up with experimental results that ground the discussion in measured performance.

## IVFPQ Recap

ScaNN was proposed by Google in 2020, and the paper reports a 3× performance improvement over HNSW on the GloVe dataset. You can refer to the [original paper](https://arxiv.org/pdf/1908.10396.pdf) and the [open-source implementation](https://github.com/google-research/google-research/tree/master/scann) for details. 

Before introducing ScaNN, we’ll briefly recap IVFPQ, since ScaNN is built on top of the same overall framework.

**IVFPQ stands for Inverted File with Product Quantization**, an algorithm used for efficient and large-scale Approximate Nearest Neighbor (ANN) search in high-dimensional vector databases. It is a hybrid approach that combines two techniques, the **inverted file index (IVF)** and **product quantization (PQ)**, to balance search speed, memory usage, and accuracy. 

### How IVFPQ Works

The process involves two main steps during indexing and searching: 

- IVF layer: vectors are clustered into `nlist` inverted lists (clusters). At query time, you visit only a subset of clusters (`nprobe`) to trade off recall and latency/throughput.

![](https://assets.zilliz.com/ivf1_5e3d29c392.png)

- PQ layer: within each visited cluster, each D-dimensional vector is split into m subvectors, each of dimension (D/m). Each subvector is quantized by assigning it to the nearest centroid in its sub-codebook. If a sub-codebook has 256 centroids, each subvector can be represented by a `uint8` code (an ID in [0, 255]).

![](https://assets.zilliz.com/pq2_6695c9cc6f.png)

Distance computation can then be rewritten as the sum over subvectors:

**D(q, X) = D(q, u0) + D(q, u1) + D(q, u2) + ... + D(q, un)**

**= L(q, id1) + L(q, id2) + L(q, id3) + ... + L(q, idn)**

Here, L represents a lookup table. At query time, the lookup table is constructed, recording the distance between the query and each quantized subvector. All subsequent distance computations are converted into table lookups followed by summation.

![](https://assets.zilliz.com/query_vector3_75d47bdd53.png)

For example, for 128-dimensional vectors split into 32 subvectors of 4 dimensions each, if each subvector is encoded by a `uint8` ID, the storage cost per vector drops from (128 x 4) bytes to (32 x 1) bytes—a 1/16 reduction.

## ScaNN Optimizations Based on IVFPQ

In summary, ScaNN improves IVFPQ in two aspects:

1.  Quantization: ScaNN proposes an objective beyond simply replacing each subvector with its nearest k-means centroid (i.e., minimizing reconstruction error).
    
2.  Lookup efficiency: ScaNN accelerates LUT-based search, which is often memory-bound, via a SIMD-friendly FastScan path.
    

### Score-aware Quantization Loss

![](https://assets.zilliz.com/text_765e538864.png) 

![](https://assets.zilliz.com/score_aware_quantization_loss_d9d36223c2.png)

![](https://assets.zilliz.com/text2_dd95418ba9.png)

![](https://assets.zilliz.com/quantizer_f0e39762a4.png)

Since the analysis is based on the IP metric, after ScaNN decomposes the quantization error into parallel and perpendicular components, only the parallel component affects the result, so a larger penalty term should be applied. Consequently, the loss function can be rewritten as follows:

$$\begin{aligned} \ell\left(x_i, \tilde{x}_i, w\right) &=h_{\|}\left(w,\left\|x_i\right\|\right)\left\|r_{\|}\left(x_i, \tilde{x}_i\right)\right\|^2 +h_{\perp}\left(w,\left\|x_i\right\|\right)\left\|r_{\perp}\left(x_i, \tilde{x}_i\right)\right\|^2 \end{aligned}$$

The figure below shows a two-dimensional example illustrating that the error caused by the parallel component is larger and can lead to incorrect nearest neighbor results, thus warranting a more severe penalty.

![The left figure shows poor quantization because the parallel offset affects the final result, while the right figure shows better quantization.](https://assets.zilliz.com/quantization_cp_81e23dd1df.png)

### 4-bit PQ FastScan

Let's first review the PQ computation process: during querying, the distances between the query and subvector cluster centers are pre-computed to construct a lookup table. Distance computation is then performed through table lookups to obtain segment distances and sum them.

![](https://assets.zilliz.com/PQ_Fast_Scan1_248c53bc93.png)

However, frequent memory reads still become a performance bottleneck. If the lookup table can be made small enough to fit in registers, memory read operations can be transformed into efficient CPU SIMD instructions.

First, each subvector is clustered into 16 classes, so a 4-bit value can represent a cluster center—this is the origin of the name "4-bit PQ." Then, distances typically represented as floats are further converted to uint8 using Scalar Quantization (SQ). This way, the lookup table for one subvector can be stored in a register using 16 × 8 = 128 bits.

![](https://assets.zilliz.com/PQ_Fast_Scan2_07b9589195.png)

Finally, let's examine the register storage layout (using AVX2 instruction set as an example): the subvectors of 32 vectors are placed in a 128-bit register, combined with the lookup table. The "lookup" operation can then be efficiently completed using a single SIMD shuffle CPU instruction.

![register layout](https://assets.zilliz.com/register_layout_b2bf8c2b2b.png)

![SIMD Shuffle for Lookup](https://assets.zilliz.com/SIMD_Shuffle_for_Lookup_c86e6d9657.png)

Here's an interesting observation: the ScaNN paper focuses entirely on the first optimization, which is reasonable since it can be considered an algorithm paper emphasizing mathematical derivations. However, the experimental results presented in the paper are remarkably impressive.

![The experimental results presented in the ScaNN paper.](https://assets.zilliz.com/experimental_results_a46ec830a7.png)

Intuitively, optimizing the loss alone should not produce such dramatic effects. Another [blog](https://medium.com/@kumon/similarity-search-scann-and-4-bit-pq-fastscan-ab98766b32bd) has also pointed this out—what really makes the difference is the 4-bit PQ FastScan portion.

## Experimental Results

Using the vector database benchmark tool [VectorDBBench](https://github.com/zilliztech/VectorDBBench), we conducted a simple test. ScaNN's performance advantage over traditional IVFFLAT and IVF_PQ is quite evident. After integration into Milvus, on the Cohere1M dataset at the same recall rate, QPS can reach 5x that of IVFFLAT and 6x that of IVF_PQ.

However, QPS is slightly lower than that of graph-based indexes like HNSW, so it is not the first choice for high-QPS use cases. But for scenarios with lower recall, it is acceptable (such as in some recommendation systems), using ScaNN without loading raw data can achieve impressive QPS with an extremely low memory footprint (1/16 of the original data), making it an excellent index choice.

| Index_Type | Case | QPS | latency(p99) | recall | memory |
| --- | --- | --- | --- | --- | --- |
| IVFFLAT | Performance1M | 266 | 0.0173s | 0.9544 | 3G |
| HNSW | Performance1M | 1885 | 0.0054s | 0.9438 | 3.24G |
| IVF_PQ | Performance1M | 208 | 0.0292s | 0.928 | 0.375G |
| ScaNN（with_raw_data: true） | Performance1M | 1215 | 0.0069s | 0.9389 | 3.186G |
| ScaNN（with_raw_data: false） | Performance1M | 1265 | 0.0071s | 0.7066 | 0.186G |

## Conclusion

ScaNN builds on the familiar IVFPQ framework but pushes it significantly further through deep engineering work in both quantization and low-level lookup acceleration. By aligning the quantization objective with ranking quality and eliminating memory bottlenecks in the inner loop, ScaNN combines score-aware quantization with a 4-bit PQ FastScan path that turns a traditionally memory-bound process into an efficient, SIMD-friendly computation.

In practice, this gives ScaNN a clear and well-defined niche. It is not intended to replace graph-based indexes like HNSW in high-recall settings. Instead, for recall-insensitive workloads with tight memory budgets, ScaNN delivers high throughput with a very small footprint. In our experiments, after integration into Milvus VectorDB, ScaNN achieved roughly 5× the QPS of IVFFLAT on the Cohere1M dataset, making it a strong choice for high-throughput, low-latency ANN retrieval where compression and efficiency matter more than perfect recall.

If you’re interested in exploring ScaNN further or discussing index selection in real-world systems, join our [Slack Channel](https://milvusio.slack.com/join/shared_invite/zt-3nntzngkz-gYwhrdSE4~76k0VMyBfD1Q#/shared-invite/email) to chat with our engineers and other AI engineers in the community. You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


A Brief Introduction to the ScaNN Index and How It Improves IVFPQ


A Brief Introduction to the ScaNN Index 




## The Problem: RAG Noise and Token Waste

**Vector search** is a solid foundation for RAG systems—enterprise assistants, AI agents, customer support bots, and more. It reliably finds the documents that matter. But retrieval alone doesn’t solve the context problem. Even well-tuned indexes return chunks that are broadly relevant, while only a small fraction of the sentences inside those chunks actually answer the query.

In production systems, this gap shows up immediately. A single query may pull in dozens of documents, each thousands of tokens long. Only a handful of sentences contain the actual signal; the rest is context that bloats token usage, slows inference, and often distracts the LLM. The problem becomes even more obvious in agent workflows, where the queries themselves are the output of multi-step reasoning and only match small parts of the retrieved text.

This creates a clear need for a model that can _**identify and highlight**_ _the useful sentences and ignore the rest_—essentially, sentence-level relevance filtering, or what many teams refer to as [**context pruning**](https://milvus.io/blog/llm-context-pruning-a-developers-guide-to-better-rag-and-agentic-ai-results.md). The goal is simple: keep the parts that matter and drop the noise before it ever reaches the LLM.

Traditional keyword-based highlighting can’t solve this problem. For example, if a user asks, “How do I improve Python code execution efficiency?”, a keyword highlighter will pick out “Python” and “efficiency,” but miss the sentence that actually answers the question—“Use NumPy vectorized operations instead of loops”—because it shares no keywords with the query. What we need instead is semantic understanding, not string matching.

## A Semantic Highlighting Model for RAG Noise Filtering and Context Pruning 

To make this easy for RAG builders, we trained and open-sourced a [**Semantic Highlighting model**](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1) that identifies and highlights the sentences in retrieved documents that are more semantically aligned with the query. The model currently delivers the state-of-the-art performance on both English and Chinese and is designed to slot directly into existing RAG pipelines.

![](https://assets.zilliz.com/context_pruning_80f7b16280.png)

**Model Details**

*   **HuggingFace:** [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1)
    
*   **License:** MIT (commercial-friendly)
    
*   **Architecture:** 0.6B encoder-only model based on BGE-M3 Reranker v2
    
*   **Context Window:** 8192 tokens
    
*   **Supported Languages:** English and Chinese
    

Semantic Highlighting provides the relevance signals needed to select only the useful parts of long retrieved documents. In practice, this model enables:

*   **Improved interpretability**, showing which parts of a document actually matter
    
*   **70–80% reduction in token cost** by sending only highlighted sentences to the LLM
    
*   **Better answer quality**, since the model sees less irrelevant context
    
*   **Easier debugging**, because engineers can inspect sentence-level matches directly
    

### Evaluation Results: Achieving SOTA Performance

We evaluated our Semantic Highlighting model across multiple datasets spanning both English and Chinese, in both in-domain and out-of-domain conditions. 

The benchmark suites include:

*   **English multi-span QA:** multispanqa
    
*   **English out-of-domain Wikipedia:** wikitext2
    
*   **Chinese multi-span QA:** multispanqa_zh
    
*   **Chinese out-of-domain Wikipedia:** wikitext2_zh
    

![](https://assets.zilliz.com/benchmarking_results_25545c952f.png)

Evaluated models include:

*   Open Provence series
    
*   Naver's Provence/XProvence series
    
*   OpenSearch's semantic-highlighter
    
*   Our trained bilingual model: [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1)
    

Across all four datasets, our model achieves the top ranking. More importantly, it is the _only_ model that performs consistently well on both English and Chinese. Competing models either focus exclusively on English or show clear performance drops on Chinese text.

## How We Built This Semantic Highlighting Model

Training a model for this task isn’t the hard part; training a _good_ model that handles the earlier problems and delivers near-SOTA performance is where the real work happens. Our approach focused on two things:

*   **Model architecture:** use an encoder-only design for fast inference.
    
*   **Training data:** generate high-quality relevance labels using reasoning-capable LLMs and scale data generation with local inference frameworks.
    

### Model Architecture

We built the model as a lightweight **encoder-only** network that treats context pruning as a **token-level relevance scoring task**. This design is inspired by [Provence](https://arxiv.org/html/2501.16214v1), a context-pruning approach introduced by Naver at ICLR 2025, which reframes pruning from “choose the right chunk” to “score every token.” That framing aligns naturally with semantic highlighting, where fine-grained signals are essential.

Encoder-only models aren’t the newest architecture, but they remain extremely practical here: they’re fast, easy to scale, and can produce relevance scores for all token positions in parallel. For a production RAG system, that speed advantage matters far more than using a larger decoder model.

Once we compute token-level relevance scores, we aggregate them into **sentence-level** scores. This step turns noisy token signals into a stable, interpretable relevance metric. Sentences above a configurable threshold are highlighted; everything else is filtered out. This produces a simple and reliable mechanism for selecting the sentences that actually matter to the query.

### Inference Process

At runtime, our semantic highlighting model follows a simple pipeline:

1.  **Input—** The process starts with a user query. Retrieved documents are treated as candidate context for relevance evaluation.
    
2.  **Model Processing—** The query and context are concatenated into a single sequence: [BOS] + Query + Context
    
3.  **Token Scoring—** Each token in the context is assigned a relevance score between 0 and 1, reflecting how strongly it is related to the query.
    
4.  **Sentence Aggregation—** Token scores are aggregated at the sentence level, typically by averaging, to produce a relevance score for each sentence.
    
5.  **Threshold Filtering—** Sentences with scores above a configurable threshold are highlighted and retained, while low-scoring sentences are filtered out before being passed to the downstream LLM.
    

![](https://assets.zilliz.com/semantic_highlighting_workflows_db3d12a666.png)

### Base Model: BGE-M3 Reranker v2

We selected BGE-M3 Reranker v2 as our base model for several reasons:

1.  It employs an Encoder architecture suitable for token and sentence scoring
    
2.  Supports multiple languages with optimization for both English and Chinese
    
3.  Provides an 8192-token context window appropriate for longer RAG documents
    
4.  Maintains 0.6B parameters—strong enough without being computationally heavy
    
5.  Ensures sufficient world knowledge in the base model
    
6.  Trained for reranking, which closely aligns with relevance judgment tasks
    

## Training Data: LLM Annotation with Reasoning

Once we finalized the model architecture, the next challenge was building a dataset that would actually train a reliable model. We started by looking at how Open Provence handles this. Their approach uses public QA datasets and a small LLM to label which sentences are relevant. It scales well and is easy to automate, which made it a good baseline for us.

But we quickly ran into the same issue they describe: if you ask an LLM to output sentence-level labels directly, the results aren’t always stable. Some labels are correct, others are questionable, and it’s hard to clean things up afterward. Fully manual annotation wasn’t an option either—we needed far more data than we could ever label by hand.

To improve stability without sacrificing scalability, we made one change: the LLM must provide a short reasoning snippet for every label it outputs. Each training example includes the query, the document, the sentence spans, and a brief explanation of why a sentence is relevant or irrelevant. This small adjustment made the annotations much more consistent and gave us something concrete to reference when validating or debugging the dataset.

Including the reasoning turned out to be surprisingly valuable:

*   **Higher annotation quality:** Writing out the reasoning works as a self-check, which reduces random or inconsistent labels.
    
*   **Better observability:** We can see _why_ a sentence was selected instead of treating the label as a black box.
    
*   **Easier debugging:** When something looks wrong, the reasoning makes it easy to spot whether the issue is the prompt, the domain, or the annotation logic.
    
*   **Reusable data:** Even if we switch to a different labeling model in the future, the reasoning traces remain useful for re-labeling or auditing.
    

The annotation workflow looks like this:

![](https://assets.zilliz.com/annotation_data_generation_ff93eb18f4.png)

### Qwen3 8B for Annotation

For annotation, we chose Qwen3 8B because it natively supports a “thinking mode” via outputs, making it much easier to extract consistent reasoning traces. Smaller models didn’t give us stable labels, and larger models were slower and unnecessarily expensive for this kind of pipeline. Qwen3 8B hit the right balance between quality, speed, and cost.

We ran all annotations using a **local vLLM service** instead of cloud APIs. This gave us high throughput, predictable performance, and much lower cost—essentially trading GPU time for API token fees, which is the better deal when generating millions of samples.

### Dataset Scale

In total, we built **over 5 million bilingual training samples**, split roughly evenly between English and Chinese.

*   **English sources:** MS MARCO, Natural Questions, GooAQ
    
*   **Chinese sources:** DuReader, Chinese Wikipedia, mmarco_chinese
    

Part of the dataset comes from re-annotating existing data used by projects like Open Provence. The rest was generated from raw corpora by first creating query–context pairs and then labeling them with our reasoning-based pipeline.

All annotated training data is also available on HuggingFace for community development and training reference: [Zilliz Datasets](https://huggingface.co/zilliz/datasets)

![](https://assets.zilliz.com/zilliz_datasets_dd91330d4d.png)

### Training Method

Once the model architecture and dataset were ready, we trained the model on **8× A100 GPUs** for three epochs, which took roughly **9 hours** end-to-end. 

**Note:** The training only targeted the **Pruning Head**, which is responsible for the semantic highlighting task. We did not train the **Rerank Head**, since focusing solely on the pruning objective yielded better results for sentence-level relevance scoring.

## Real-World Case Study

Benchmarks only tell part of the story, so here’s a real example that shows how the model behaves on a common edge case: when the retrieved text contains both the correct answer and a very tempting distractor.

**Query:** _Who wrote The Killing of a Sacred Deer?_

**Context (5 sentences):**


```
1\. The Killing of a Sacred Deer is a 2017 psychological horror film directed by Yorgos Lanthimos,

   with a screenplay by Lanthimos and Efthymis Filippou.

2\. The film stars Colin Farrell, Nicole Kidman, Barry Keoghan, Raffey Cassidy,

   Sunny Suljic, Alicia Silverstone, and Bill Camp.

3\. The story is based on the ancient Greek playwright Euripides' play Iphigenia in Aulis.

4\. The film tells the story of a cardiac surgeon (Farrell) who secretly

   befriends a teenager (Keoghan) connected to his past.

5\. He introduces the boy to his family, who then mysteriously fall ill.
```

Correct answer: Sentence 1 (explicitly states "screenplay by Lanthimos and Efthymis Filippou")

This example has a trap: Sentence 3 mentions that "Euripides" wrote the original play. But the question asks "who wrote the film The Killing of a Sacred Deer," and the answer should be the film's screenwriters, not the Greek playwright from thousands of years ago.

### Model results


| Model        | Finds correct answer? | Prediction                                      |
| ------------ | --------------------- | ----------------------------------------------- |
| Our Model    | ✓                     | Selected sentences 1 (correct) and 3            |
| XProvence v1 | ✗                     | Only selected sentence 3, missed correct answer |
| XProvence v2 | ✗                     | Only selected sentence 3, missed correct answer |


**Key Sentence Score Comparison:**

| Sentence                                     | Our Model | XProvence v1 | XProvence v2 |
| -------------------------------------------- | --------- | ------------ | ------------ |
| Sentence 1 (film screenplay, correct answer) | 0.915     | 0.133        | 0.081        |
| Sentence 3 (original play, distractor)       | 0.719     | 0.947        | 0.802        |


XProvence models:


*   Strongly attracted to "Euripides" and "play," giving sentence 3 near-perfect scores (0.947 and 0.802)
    
*   Completely ignores the actual answer (sentence 1), giving extremely low scores (0.133 and 0.081)
    
*   Even when lowering the threshold from 0.5 to 0.2, it still can't find the correct answer
    

Our model:

*   Correctly gives sentence 1 the highest score (0.915)
    
*   Still assigns sentence 3 some relevance (0.719) because it’s related to the background
    
*   Clearly separates the two with a ~0.2 margin
    

This example shows the model’s core strength: understanding **query intent** rather than just matching surface-level keywords. In this context, “Who wrote _The Killing of a Sacred Deer_” refers to the film, not the ancient Greek play. Our model picks up on that, while others get distracted by strong lexical cues.

## Try It Out and Tell Us What You Think

Our [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1) model is now fully open-sourced under the MIT license and ready for production use. You can plug it into your RAG pipeline, fine-tune it for your own domain, or build new tools on top of it. We also welcome contributions and feedback from the community.

*   **Download from HuggingFace**: [zilliz/semantic-highlight-bilingual-v1](https://huggingface.co/zilliz/semantic-highlight-bilingual-v1)
    
*   **All annotated training data**: [https://huggingface.co/zilliz/datasets](https://huggingface.co/zilliz/datasets) 
    

### Semantic Highlighting Available in Milvus and Zilliz Cloud

Semantic highlighting is also built directly into [Milvus](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/cloud) (the fully managed Milvus), giving users a clear view of _why_ each document was retrieved. Instead of scanning entire chunks, you immediately see the specific sentences that relate to your query — even when the wording doesn’t match exactly. This makes retrieval easier to understand and much faster to debug. For RAG pipelines, it also clarifies what the downstream LLM is expected to focus on, which helps with prompt design and quality checks.

[**Try Semantic Highlighting in a fully managed Zilliz Cloud for free**](https://cloud.zilliz.com/signup?utm_source=milvusio&utm_page=semantic-highlighting-blog) 

We’d love to hear how it works for you—bug reports, improvement ideas, or anything you discover while integrating it into your workflow.

If you want to talk through anything in more detail, feel free to join our [Discord channel](https://discord.com/invite/8uyFbECzPX) or book a 20-minute [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) session. We’re always happy to chat with other builders and swap notes.

## Acknowledgements

This work builds on a lot of great ideas and open-source contributions, and we want to highlight the projects that made this model possible.

*   **Provence** introduced a clean and practical framing for context pruning using lightweight encoder models.
    
*   **Open Provence** provided a solid, well-engineered codebase—training pipelines, data processing, and model heads—under a permissive license. It gave us a strong starting point for experimentation.
    

On top of that foundation, we added several contributions of our own:

*   Using **LLM reasoning** to generate higher-quality relevance labels
    
*   Creating **nearly 5 million** bilingual training samples aligned with real RAG workloads
    
*   Choosing a base model better suited for long-context relevance scoring (**BGE-M3 Reranker v2**)
    
*   Training only the **Pruning Head** to specialize the model for semantic highlighting
    

We’re grateful to the Provence and Open Provence teams for publishing their work openly. Their contributions significantly accelerated our development and made this project possible.


Semantic Highlighting for RAG Context Pruning and Token Saving


How We Built a Semantic Highlighting Model for RAG Context Pruning and Token Saving


LLM Context Pruning: Improving RAG and Agentic AI Systems


LLM Context Pruning: A Developer’s Guide to Better RAG and Agentic AI Results



_This post is written by Min Yin, one of the most active community contributors of Milvus, and is published here with permission._ 


[MinIO](https://github.com/minio/minio) is an open-source, high-performance, and S3-compatible object storage system widely used in AI/ML, analytics, and other data-intensive workloads. For many [Milvus](https://milvus.io/) deployments, it has also been the default choice for object storage. Recently, however, the MinIO team updated its [GitHub README](https://github.com/minio/minio?tab=readme-ov-file) to state that **_this project is no longer accepting new changes_**_._

![](https://assets.zilliz.com/minio_7b7df16860.png)

Actually, over the past few years, MinIO has gradually shifted its attention toward commercial offerings, tightened its licensing and distribution model, and scaled back active development in the community repository. Moving the open-source project into maintenance mode is the natural outcome of that broader transition.

For Milvus users who rely on MinIO by default, this change is hard to ignore. Object storage sits at the heart of Milvus’s persistence layer, and its reliability over time depends not just on what works today but on whether the system continues to evolve alongside the workloads it supports.

Against this backdrop, this article explores [RustFS](https://github.com/rustfs/rustfs) as a potential alternative. RustFS is a Rust-based, S3-compatible object storage system that emphasizes memory safety and modern systems design. It is still experimental, and this discussion is not a production recommendation. 


## The Milvus Architecture and Where the Object Storage Component Sits

Milvus 2.6 adopts a fully decoupled storage–compute architecture. In this model, the storage layer is composed of three independent components, each serving a distinct role.

Etcd stores metadata, Pulsar or Kafka handles streaming logs, and object storage—typically MinIO or an S3-compatible service—provides durable persistence for vector data and index files. Because storage and compute are separated, Milvus can scale compute nodes independently while relying on a shared, reliable storage backend.

![](https://assets.zilliz.com/milvus_architecture_fe897f1098.webp)

### The Role of Object Storage in Milvus

Object storage is the durable storage layer in Milvus. Raw vector data is persisted as binlogs, and index structures such as HNSW and IVF_FLAT are stored there as well. 

This design makes compute nodes stateless. Query Nodes do not store data locally; instead, they load segments and indexes from object storage on demand. As a result, nodes can scale up or down freely, recover quickly from failures, and support dynamic load balancing across the cluster without data rebalancing at the storage layer.

```
my-milvus-bucket/
├── files/                          # rootPath (default)
│   ├── insert_log/                 # insert binlogs
│   │   └── {Collection_ID}/
│   │       └── {Partition_ID}/
│   │           └── {Segment_ID}/
│   │               └── {Field_ID}/
│   │                   └── {Log_ID}     # Per-field binlog files
│   │
│   ├── delta_log/                  # Delete binlogs
│   │   └── {Collection_ID}/
│   │       └── {Partition_ID}/
│   │           └── {Segment_ID}/
│   │               └── {Log_ID}        
│   │
│   ├── stats_log/                  # Statistical data (e.g., Bloom filters)
│   │   └── {Collection_ID}/
│   │       └── {Partition_ID}/
│   │           └── {Segment_ID}/
│   │               └── {Field_ID}/
│   │                   └── {Log_ID}
│   │
│   └── index_files/                # Index files
│       └── {Build_ID}_{Index_Version}_{Segment_ID}_{Field_ID}/
│           ├── index_file_0
│           ├── index_file_1
│           └── ...
```

### Why Milvus Uses the S3 API

Rather than defining a custom storage protocol, Milvus uses the S3 API as its object storage interface. S3 has become the de facto standard for object storage: major cloud providers such as AWS S3, Alibaba Cloud OSS, and Tencent Cloud COS support it natively, while open-source systems like MinIO, Ceph RGW, SeaweedFS, and RustFS are fully compatible.

By standardizing on the S3 API, Milvus can rely on mature, well-tested Go SDKs instead of maintaining separate integrations for each storage backend. This abstraction also gives users deployment flexibility: MinIO for local development, cloud object storage in production, or Ceph and RustFS for private environments. As long as an S3-compatible endpoint is available, Milvus does not need to know—or care—what storage system is used underneath.

```
# Milvus configuration file milvus.yaml
minio:
 address: localhost
 port: 9000
 accessKeyID: minioadmin
 secretAccessKey: minioadmin
 useSSL: false
 bucketName: milvus-bucket
```

## Evaluating RustFS as an S3-Compatible Object Storage Backend for Milvus

### Project Overview

RustFS is a distributed object storage system written in Rust. It is currently in the alpha stage (version 1.0.0-alpha.68) and aims to combine the operational simplicity of MinIO with Rust’s strengths in memory safety and performance. More details are available on [GitHub](https://github.com/rustfs/rustfs).

RustFS is still under active development, and its distributed mode has not yet been officially released. As a result, RustFS is not recommended for production or mission-critical workloads at this stage.


### Architecture Design

RustFS follows a design that is conceptually similar to MinIO. An HTTP server exposes an S3-compatible API, while an Object Manager handles object metadata, and a Storage Engine is responsible for data block management. At the storage layer, RustFS relies on standard file systems such as XFS or ext4.

For its planned distributed mode, RustFS intends to use etcd for metadata coordination, with multiple RustFS nodes forming a cluster. This design aligns closely with common object storage architectures, making RustFS familiar to users with MinIO experience.

![](https://assets.zilliz.com/architecture_design_852f73b2c8.png)

### Compatibility with Milvus

Before considering RustFS as an alternative object storage backend, it is necessary to evaluate whether it meets Milvus’s core storage requirements.


|  **Milvus Requirement**  |           **S3 API**          |         **RustFS Support**         |
| :----------------------: | :---------------------------: | :--------------------------------: |
|  Vector data persistence |    `PutObject`, `GetObject`   |          ✅ Fully supported         |
|   Index file management  | `ListObjects`, `DeleteObject` |          ✅ Fully supported         |
| Segment merge operations |        Multipart Upload       |          ✅ Fully supported         |
|  Consistency guarantees  |    Strong read-after-write    | ✅ Strong consistency (single-node) |

Based on this evaluation, RustFS’s current S3 API implementation satisfies Milvus’s baseline functional requirements. This makes it suitable for practical testing in non-production environments.


## Hands-On: Replacing MinIO with RustFS in Milvus

The goal of this exercise is to replace the default MinIO object storage service and deploy Milvus 2.6.7 using RustFS as the object storage backend.


### Prerequisites

1. Docker and Docker Compose are installed (version ≥ 20.10), and the system can pull images and run containers normally.

2. A local directory is available for object data storage, such as `/volume/data/` (or a custom path).

3. The host port 9000 is open for external access, or an alternative port is configured accordingly.

4. The RustFS container runs as a non-root user (`rustfs`). Ensure the host data directory is owned by UID 10001.


### Step 1: Create the Data Directory and Set Permissions

```
# Create the project directory
mkdir -p milvus-rustfs && cd milvus-rustfs
# Create the data directory
mkdir -p volumes/{rustfs, etcd, milvus}
# Update permissions for the RustFS directory
sudo chown -R 10001:10001 volumes/rustfs
```

**Download the Official Docker Compose File**

```
wget https://github.com/milvus-io/milvus/releases/download/v2.6.7/milvus-standalone-docker-compose.yml -O docker-compose.yml
```

### Step 2: Modify the Object Storage Service

**Define the RustFS Service**

Note: Make sure the access key and secret key match the credentials configured in the Milvus service.

```
rustfs:
 container_name: milvus-rustfs
 image: registry.cn-hangzhou.aliyuncs.com/rustfs/rustfs: latest
 environment:
 RUSTFS_ACCESS_KEY: minioadmin
 RUSTFS_SECRET_KEY: minioadmin
 RUSTFS_CONSOLE_ENABLE: “true”
 RUSTFS_REGION: us-east-1
 # RUSTFS_SERVER_DOMAINS: localhost# Optional; not required for local deployments
 ports:
 - “9001:9001”
 - “9000:9000”
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/rustfs:/data
 command: >
 --address :9000
 --console-enable
 /data
 healthcheck:
 test: [“CMD”, “curl”, “-f”, “http://localhost:9000/health"]
 interval: 30s
 timeout: 20s
 retries: 3
```

**Complete Configuration**

Note: Milvus’s storage configuration currently assumes MinIO-style defaults and does not yet allow custom access key or secret key values. When using RustFS as a replacement, it must use the same default credentials expected by Milvus.

```
version: ‘3.5’
services:
 etcd:
 container_name: milvus-etcd
 image: registry.cn-hangzhou.aliyuncs.com/etcd/etcd: v3.5.25
 environment:
 - ETCD_AUTO_COMPACTION_MODE=revision
 - ETCD_AUTO_COMPACTION_RETENTION=1000
 - ETCD_QUOTA_BACKEND_BYTES=4294967296
 - ETCD_SNAPSHOT_COUNT=50000
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
 command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
 healthcheck:
 test: [“CMD”, “etcdctl”, “endpoint”, “health”]
 interval: 30s
 timeout: 20s
 retries: 3
 rustfs:
 container_name: milvus-rustfs
 image: registry.cn-hangzhou.aliyuncs.com/rustfs/rustfs: latest
 environment:
 RUSTFS_ACCESS_KEY: minioadmin
 RUSTFS_SECRET_KEY: minioadmin
 RUSTFS_CONSOLE_ENABLE: “true”
 RUSTFS_REGION: us-east-1
 # RUSTFS_SERVER_DOMAINS: localhost# Optional; not required for local deployments
 ports:
 - “9001:9001”
 - “9000:9000”
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/rustfs:/data
 command: >
 --address :9000
 --console-enable
 /data
 healthcheck:
 test: [“CMD”, “curl”, “-f”, “http://localhost:9000/health"]
 interval: 30s
 timeout: 20s
 retries: 3
 standalone:
 container_name: milvus-standalone
 image: registry.cn-hangzhou.aliyuncs.com/milvus/milvus: v2.6.7
 command: [”milvus“, ”run“, ”standalone“]
 security_opt:
 - seccomp: unconfined
 environment:
 MINIO_REGION: us-east-1
 ETCD_ENDPOINTS: etcd:2379
 MINIO_ADDRESS: rustfs:9000
 MINIO_ACCESS_KEY: minioadmin
 MINIO_SECRET_KEY: minioadmin
 MINIO_USE_SSL: ”false“
 MQ_TYPE: rocksmq
 volumes:
 - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
 healthcheck:
 test: [”CMD“, ”curl“, ”-f“, ”http://localhost:9091/healthz"]
 interval: 30s
 start_period: 90s
 timeout: 20s
 retries: 3
 ports:
 - “19530:19530”
 - “9091:9091”
 depends_on:
 - “etcd”
 - “rustfs”
networks:
 default:
 name: milvus
```

**Start the Services**

```
docker-compose -f docker-compose.yaml up -d
```

**Check Service Status**

```
docker-compose ps -a
```

![](https://assets.zilliz.com/code_d64dc88a96.png)

**Access the RustFS Web UI**

Open the RustFS web interface in your browser: <http://localhost:9001>

Log in using the default credentials: username and password are both minioadmin.

**Test the Milvus Service**

```
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
# connect to Milvus
connections.connect(
 alias="default",
 host='localhost',
 port='19530'
)
print("✓ Successfully connected to Milvus!")
# create test collection
fields = [
 FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
 FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128)
]
schema = CollectionSchema(fields=fields, description="test collection")
collection = Collection(name="test_collection", schema=schema)
print("✓ Test collection created!")
print("✓ RustFS verified as the S3 storage backend!")

### Step 3: Storage Performance Testing (Experimental)

**Test Design**

Two Milvus deployments were set up on identical hardware (16 cores / 32 GB memory / NVMe SSD), using RustFS and MinIO respectively as the object storage backend. The test dataset consisted of 1,000,000 vectors with 768 dimensions, using an HNSW index with parameters _M = 16_ and _efConstruction = 200_. Data was inserted in batches of 5,000.

The following metrics were evaluated: Insert throughput, Index build time, Cold and warm load time, Search latency, Storage footprint.

**Test Code**

Note: Only the core parts of the test code are shown below.

```
def milvus_load_bench(dim=768, rows=1_000_000, batch=5000):
    collection = Collection(...)
    # Insert test
    t0 = time.perf_counter()
    for i in range(0, rows, batch):
        collection.insert([rng.random((batch, dim), dtype=np.float32).tolist()])
    insert_time = time.perf_counter() - t0
    # Index build
    collection.flush()
    collection.create_index(field_name="embedding", 
                           index_params={"index_type": "HNSW", ...})
    # Load test (cold start + two warm starts)
    collection.release()
    load_times = []
    for i in range(3):
        if i > 0: collection.release(); time.sleep(2)
        collection.load()
        load_times.append(...)
    # Query test
    search_times = []
    for _ in range(3):
        collection.search(queries, limit=10, ...)
```

**Test Command**

```
# RustFS: --port 19530 --s3-endpoint http://localhost:9000 --s3-bucket bench
# MinIO: --port 19531 --s3-endpoint http://localhost:9001 --s3-bucket a-bucket
python bench.py milvus-load-bench --dim 768 --rows 1000000 --batch 5000 \
  --index-type HNSW --repeat-load 3 --release-before-load --do-search --drop-after
```

**Performance Results**

- **RustFS**

Faster writes (+57%), lower storage usage (–57%), and faster warm loads (+67%), making it suitable for write-heavy, cost-sensitive workloads. 

Much slower queries (7.96 ms vs. 1.85 ms, ~+330% latency) with noticeable variance (up to 17.14 ms), and 43% slower index builds. Not suitable for query-intensive applications.

- **MinIO**

Excellent query performance (**1.85 ms** average latency), mature small-file and random I/O optimizations, and a well-established ecosystem.


|     **Metric**    |  **RustFS**  |   **MinIO**  | **Difference** |
| :---------------: | :----------: | :----------: | :------------: |
| Insert Throughput | 4,472 rows/s | 2,845 rows/s |      0.57      |
|  Index Build Time |     803 s    |     562 s    |      -43%      |
| Load (Cold Start) |    22.7 s    |    18.3 s    |      -24%      |
| Load (Warm Start) |    0.009 s   |    0.027 s   |      0.67      |
|   Search Latency  |    7.96 ms   |    1.85 ms   |    **-330%**   |
|   Storage Usage   |    7.8 GB    |    18.0 GB   |      0.57      |

RustFS significantly outperforms MinIO in write performance and storage efficiency, with both showing roughly 57% improvement. This demonstrates the system-level advantages of the Rust ecosystem. However, the 330% gap in query latency limits RustFS’s suitability for query-intensive workloads.

For **production environments**, cloud-managed object storage services like **AWS S3** are recommended first, as they are mature, stable, and require no operational effort. Self-hosted solutions are better suited to specific scenarios: RustFS for cost-sensitive or write-intensive workloads, MinIO for query-intensive use cases, and Ceph for data sovereignty. With further optimization in random read performance, RustFS has the potential to become a strong self-hosted option.


## Conclusion

MinIO’s decision to stop accepting new community contributions is disappointing for many developers, but it won’t disrupt Milvus users. Milvus depends on the S3 API—not any specific vendor implementation—so swapping storage backends is straightforward. This S3-compatibility layer is intentional: it ensures Milvus stays flexible, portable, and decoupled from vendor lock-in.

For production deployments, cloud-managed services such as AWS S3 and Alibaba Cloud OSS remain the most reliable options. They’re mature, highly available, and drastically reduce the operational load compared to running your own object storage. Self-hosted systems like MinIO or Ceph still make sense in cost-sensitive environments or where data sovereignty is non-negotiable, but they require significantly more engineering overhead to operate safely at scale.

RustFS is interesting—Apache 2.0-licensed, Rust-based, and community-driven—but it's still early. The performance gap is noticeable, and the distributed mode hasn’t shipped yet. It’s not production-ready today, but it’s a project worth watching as it matures.

If you’re comparing object storage options for Milvus, evaluating MinIO replacements, or weighing the operational trade-offs of different backends, we’d love to hear from you.

Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) and share your thoughts. You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Evaluating RustFS for Milvus S3-Compatible Object Storage


MinIO Stops Accepting Community Changes: Evaluating RustFS as a Viable S3-Compatible Object Storage Backend for Milvus 



In high-quality AI agent systems, memory design is far more complex than it first appears. At its core, it must answer three fundamental questions: How should conversation history be stored? When should past context be retrieved? And what, exactly, should be retrieved?

These choices directly shape an agent’s response latency, resource usage, and—ultimately—its capability ceiling.

Models like ChatGPT and Claude feel increasingly “memory-aware” the more we use them. They remember preferences, adapt to long-term goals, and maintain continuity across sessions. In that sense, they already function as mini AI agents. Yet beneath the surface, their memory systems are built on very different architectural assumptions.

Recent reverse-engineering analyses of [ChatGPT](https://manthanguptaa.in/posts/chatgpt_memory/)’s and [Claude’s memory mechanisms](https://manthanguptaa.in/posts/claude_memory/) reveal a clear contrast. **ChatGPT** relies on precomputed context injection and layered caching to deliver lightweight, predictable continuity. **Claude,** by contrast, adopts RAG-style, on-demand retrieval with dynamic memory updates to balance memory depth and efficiency.

These two approaches are not just design preferences—they are shaped by infrastructure capability. [**Milvus 2.6**](https://milvus.io/docs/release_notes.md#v268) introduces the combination of hybrid dense–sparse retrieval, efficient scalar filtering, and tiered storage that on-demand conversational memory requires, making selective retrieval fast and economical enough to deploy in real-world systems.

In this post, we’ll walk through how ChatGPT’s and Claude’s memory systems actually work, why they diverged architecturally, and how recent advances in systems like Milvus make on-demand conversational retrieval practical at scale.


## ChatGPT’s Memory System

Instead of querying a vector database or dynamically retrieving past conversations at inference time, ChatGPT constructs its “memory” by assembling a fixed set of context components and injecting them directly into every prompt. Each component is prepared ahead of time and occupies a known position in the prompt.

This design keeps personalization and conversational continuity intact while making latency, token usage, and system behavior more predictable. In other words, memory is not something the model searches for on the fly—it is something the system packages and hands to the model every time it generates a response.

At a high level, a complete ChatGPT prompt consists of the following layers, ordered from most global to most immediate:

[0] System Instructions

[1] Developer Instructions

[2] Session Metadata (ephemeral)

[3] User Memory (long-term facts)

[4] Recent Conversations Summary (past chats, titles + snippets)

[5] Current Session Messages (this chat)

[6] Your latest message

Among these, components [2] through [5] form the effective memory of the system, each serving a distinct role.


### Session Metadata

Session metadata represents short-lived, non-persistent information that is injected once at the beginning of a conversation and discarded when the session ends. Its role is to help the model adapt to the current usage context rather than to personalize behavior long term. 

This layer captures signals about the user’s immediate environment and recent usage patterns. Typical signals include:

- **Device information** — for example, whether the user is on mobile or desktop

- **Account attributes** — such as subscription tier (for example, ChatGPT Go), account age, and overall usage frequency

- **Behavioral metrics** — including active days over the past 1, 7, and 30 days, average conversation length, and model usage distribution (for example, 49% of requests handled by GPT-5)


### User Memory

User memory is the persistent, editable layer of memory that enables personalization across conversations. It stores relatively stable information—such as a user’s name, role or career goals, ongoing projects, past outcomes, and learning preferences—and is injected into each new conversation to preserve continuity over time.

This memory can be updated in two ways:

- **Explicit updates** occur when users directly manage memory with instructions like “remember this” or “remove this from memory.”

- **Implicit updates** occur when the system identifies information that meets OpenAI’s storage criteria—such as a confirmed name or job title—and saves it automatically, subject to the user’s default consent and memory settings.


### Recent Conversation Summary

The recent conversation summary is a lightweight, cross-session context layer that preserves continuity without replaying or retrieving full chat histories. Instead of relying on dynamic retrieval, as in traditional RAG-based approaches, this summary is precomputed and injected directly into every new conversation.

This layer summarizes user messages only, excluding assistant replies. It is intentionally limited in size—typically around 15 entries—and retains only high-level signals about recent interests rather than detailed content. Because it does not rely on embeddings or similarity search, it keeps both latency and token consumption low.


### Current Session Messages

Current session messages contain the full message history of the ongoing conversation and provide the short-term context needed for coherent, turn-by-turn responses. This layer includes both user inputs and assistant replies, but only while the session remains active.

Because the model operates within a fixed token limit, this history cannot grow indefinitely. When the limit is reached, the system drops the earliest messages to make room for newer ones. This truncation affects only the current session: long-term user memory and the recent conversation summary remain intact.


## Claude’s Memory System

Claude takes a different approach to memory management. Rather than injecting a large, fixed bundle of memory components into every prompt—as ChatGPT does—Claude combines persistent user memory with on-demand tools and selective retrieval. Historical context is fetched only when the model judges it to be relevant, allowing the system to trade off contextual depth against computational cost.

Claude’s prompt context is structured as follows:

[0] System Prompt (static instructions)

[1] User Memories

[2] Conversation History

[3] Current Message

The key differences between Claude and ChatGPT lie in **how conversation history is retrieved** and **how user memory is updated and maintained**.


### User Memories

In Claude, user memories form a long-term context layer similar in purpose to ChatGPT’s user memory, but with a stronger emphasis on automatic, background-driven updates. These memories are stored in a structured format (wrapped in XML-style tags) and are designed to evolve gradually over time with minimal user intervention.

Claude supports two update paths:

- **Implicit updates** — The system periodically analyzes conversation content and updates memory in the background. These updates are not applied in real time, and memories associated with deleted conversations are gradually pruned as part of ongoing optimization.

- **Explicit updates** — Users can directly manage memory through commands such as “remember this” or “delete this,” which are executed via a dedicated `memory_user_edits` tool.

Compared with ChatGPT, Claude places greater responsibility on the system itself to refine, update, and prune long-term memory. This reduces the need for users to actively curate what is stored.


### Conversation History

For conversation history, Claude does not rely on a fixed summary that is injected into every prompt. Instead, it retrieves past context only when the model decides it is necessary, using three distinct mechanisms. This avoids carrying irrelevant history forward and keeps token usage under control.

|               **Component**               |                                                                     **Purpose**                                                                     |                                                              **How It’s Used**                                                             |
| :---------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: |
| **Rolling Window (Current Conversation)** |                  Stores the full message history of the current conversation (not a summary), similar to ChatGPT’s session context                  |                     Injected automatically. Token limit is ~190K; older messages are dropped once the limit is reached                    |
|       `conversation_search` **tool**      |              Searches past conversations by topic or keyword, returning conversation links, titles, and user/assistant message excerpts             | Triggered when the model determines that historical details are needed. Parameters include `query` (search terms) and `max_results` (1–10) |
|          `recent_chats` **tool**          | Retrieves recent conversations within a specified time range (for example, “past 3 days”), with results formatted the same as `conversation_search` |    Triggered when the recent, time-scoped context is relevant. Parameters include `n` (number of results), `sort_order`, and time range    |

Among these components, `conversation_search` is especially noteworthy. It can surface relevant results even for loosely phrased or multilingual queries, indicating that it operates at a semantic level rather than relying on simple keyword matching. This likely involves embedding-based retrieval, or a hybrid approach that first translates or normalizes the query into a canonical form and then applies keyword or hybrid retrieval.

Overall, Claude’s on-demand retrieval approach has several notable strengths:

- **Retrieval is not automatic**: Tool calls are triggered by the model’s own judgment. For example, when a user refers to _“the project we discussed last time,”_ Claude may decide to invoke `conversation_search` to retrieve the relevant context.

- **Richer context when needed**: Retrieved results can include **assistant response excerpts**, whereas ChatGPT’s summaries only capture user messages. This makes Claude better suited for use cases that require deeper or more precise conversational context.

- **Better efficiency by default**: Because historical context is not injected unless needed, the system avoids carrying large amounts of irrelevant history forward, reducing unnecessary token consumption.

The trade-offs are equally clear. Introducing on-demand retrieval increases system complexity: indexes must be built and maintained, queries executed, results ranked, and sometimes re-ranked. End-to-end latency also becomes less predictable than with precomputed, always-injected context. In addition, the model must learn to decide when retrieval is necessary. If that judgment fails, relevant context may never be fetched at all.


## The Constraints Behind Claude-Style On-Demand Retrieval

Adopting an on-demand retrieval model makes the vector database a critical part of the architecture. Conversation retrieval places unusually high demands on both storage and query execution, and the system must meet four constraints at the same time.

![](https://assets.zilliz.com/constraints_b6ed74e454.jpg)

### 1. Low Latency Tolerance

In conversational systems, P99 latency typically needs to stay under ~20 ms. Delays beyond that are immediately noticeable to users. This leaves little room for inefficiency: vector search, metadata filtering, and result ranking must all be carefully optimized. A bottleneck at any point can degrade the entire conversational experience.


### 2. Hybrid Search Requirement

User queries often span multiple dimensions. A request like _“discussions about RAG from the past week”_ combines semantic relevance with time-based filtering. If a database only supports vector search, it may return 1,000 semantically similar results, only for application-layer filtering to reduce them to a handful—wasting most of the computation. To be practical, the database must natively support combined vector and scalar queries.


### 3. Storage–Compute Separation

Conversation history exhibits a clear hot–cold access pattern. Recent conversations are queried frequently, while older ones are rarely touched. If all vectors had to stay in memory, storing tens of millions of conversations would consume hundreds of gigabytes of RAM—an impractical cost at scale. To be viable, the system must support storage–compute separation, keeping hot data in memory and cold data in object storage, with vectors loaded on demand.


### 4. Diverse Query Patterns

Conversation retrieval does not follow a single access pattern. Some queries are purely semantic (for example, _“the performance optimization we discussed”_), others are purely temporal (_“all conversations from last week”_), and many combine multiple constraints (_“Python-related discussions mentioning FastAPI in the last three months”_). The database query planner must adapt execution strategies to different query types, rather than relying on a one-size-fits-all, brute-force search.

Together, these four challenges define the core constraints of conversational retrieval. Any system seeking to implement Claude-style, on-demand retrieval must address all of them in a coordinated way.


## Why Milvus 2.6 Works Well for Conversational Retrieval

The design choices in [Milvus 2.6](https://milvus.io/docs/release_notes.md#v268) align closely with the core requirements of on-demand conversational retrieval. Below is a breakdown of the key capabilities and how they map to real conversational retrieval needs.

![](https://assets.zilliz.com/milvus_2_6_ce379ff42d.jpg)

### Hybrid Retrieval with Dense and Sparse Vectors

Milvus 2.6 natively supports storing dense and sparse vectors within the same collection and automatically fusing their results at query time. Dense vectors (for example, 768-dimensional embeddings generated by models like BGE-M3) capture semantic similarity, while sparse vectors (typically produced by BM25) preserve exact keyword signals. 

For a query such as _“discussions about RAG from last week,”_ Milvus executes semantic retrieval and keyword retrieval in parallel, then merges the results through reranking. Compared to using either approach alone, this hybrid strategy delivers significantly higher recall in real conversational scenarios.


### Storage–Compute Separation and Query Optimization

Milvus 2.6 supports tiered storage in two ways:

- Hot data in memory, cold data in object storage

- Indexes in memory, raw vector data in object storage

With this design, storing one million conversation entries can be achieved with roughly 2 GB of memory and 8 GB of object storage. With proper tuning, P99 latency can remain below 20 ms, even with storage–compute separation enabled.


### JSON Shredding and Fast Scalar Filtering

Milvus 2.6 enables JSON Shredding by default, flattening nested JSON fields into columnar storage. This improves scalar filtering performance by 3–5× according to official benchmarks (actual gains vary by query pattern).

Conversational retrieval often requires filtering by metadata such as user ID, session ID, or time range. With JSON Shredding, queries like _“all conversations from user A in the past week”_ can be executed directly on columnar indexes, without repeatedly parsing full JSON blobs.


### Open-Source Control and Operational Flexibility

As an open-source system, Milvus offers a level of architectural and operational control that closed, black-box solutions do not. Teams can tune index parameters, apply data tiering strategies, and customize distributed deployments to match their workloads.

This flexibility lowers the barrier to entry: small and mid-sized teams can build million- to tens-of-millions-scale conversational retrieval systems without relying on oversized infrastructure budgets.


## Why ChatGPT and Claude Took Different Paths

At a high level, the difference between ChatGPT’s and Claude’s memory systems comes down to how each handles forgetting. ChatGPT favors proactive forgetting: once memory exceeds fixed limits, older context is dropped. This trades completeness for simplicity and predictable system behavior. Claude favors delayed forgetting. In theory, conversation history can grow without bound, with recall delegated to an on-demand retrieval system.

So why did the two systems choose different paths? With the technical constraints laid out above, the answer becomes clear: **each architecture is only viable if the underlying infrastructure can support it**.

If Claude’s approach had been attempted in 2020, it would likely have been impractical. At the time, vector databases often incurred hundreds of milliseconds of latency, hybrid queries were poorly supported, and resource usage scaled prohibitively as data grew. Under those conditions, on-demand retrieval would have been dismissed as over-engineering.

By 2025, the landscape has changed. Advances in infrastructure—driven by systems such as **Milvus 2.6**—have made storage–compute separation, query optimization, dense–sparse hybrid retrieval, and JSON Shredding viable in production. These advances reduce latency, control costs, and make selective retrieval practical at scale. As a result, on-demand tools and retrieval-based memory have become not only feasible, but increasingly attractive, especially as a foundation for agent-style systems.

Ultimately, architecture choices follow what the infrastructure makes possible.


## Conclusion

In real-world systems, memory design is not a binary choice between precomputed context and on-demand retrieval. The most effective architectures are typically hybrid, combining both approaches.

A common pattern is to inject recent conversation turns through a sliding context window, store stable user preferences as fixed memory, and retrieve older history on demand via vector search. As a product matures, this balance can shift gradually—from primarily precomputed context to increasingly retrieval-driven—without requiring a disruptive architectural reset.

Even when starting with a precomputed approach, it is important to design with migration in mind. Memory should be stored with clear identifiers, timestamps, categories, and source references. When retrieval becomes viable, embeddings can be generated for existing memory and added to a vector database alongside the same metadata, allowing retrieval logic to be introduced incrementally and with minimal disruption.

Have questions or want a deep dive on any feature of the latest Milvus? Join our [Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on [GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through [Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Milvus 2.6 Makes Claude-Style On-Demand Retrieval Practical


Reflections on ChatGPT and Claude’s Memory Systems: What It Takes to Enable On-Demand Conversational Retrieval


Open-sourcing a Bilingual Semantic Highlighting Model for Production AI


We Trained and Open-Sourced a Bilingual Semantic Highlighting Model for Production RAG and AI Search



When building a database system, engineers spend most of their time on performance: index types, recall, latency, throughput, and scaling. But once a system moves beyond a single developer’s laptop, another question becomes just as critical: **who can do what inside your Milvus cluster**? In other words—access control.

Across the industry, many operational incidents stem from simple permission mistakes. A script runs against the wrong environment. A service account has broader access than intended. A shared admin credential ends up in CI. These issues usually surface as very practical questions:

- Are developers allowed to delete production collections?

- Why can a test account read production vector data?

- Why are multiple services logging in with the same admin role?

- Can analytics jobs have read-only access with zero write privileges?

[Milvus](https://milvus.io/) addresses these challenges with [role-based access control (RBAC)](https://milvus.io/docs/rbac.md). Instead of giving every user superadmin rights or trying to enforce restrictions in application code, RBAC lets you define precise permissions at the database layer. Each user or service gets exactly the capabilities it needs—nothing more.

This post explains how RBAC works in Milvus, how to configure it, and how to apply it safely in production environments.


## Why Access Control Matters When Using Milvus 

When teams are small, and their AI applications serve only a limited number of users, infrastructure is usually simple. A few engineers manage the system; Milvus is used only for development or testing; and operational workflows are straightforward. In this early stage, access control rarely feels urgent—because the risk surface is small and any mistakes can be easily reversed.

As Milvus moves into production and the number of users, services, and operators grows, the usage model changes quickly. Common scenarios include:

- Multiple business systems sharing the same Milvus instance

- Multiple teams accessing the same vector collections

- Test, staging, and production data coexisting in a single cluster

- Different roles needing different levels of access, from read-only queries to writes and operational control

Without well-defined access boundaries, these setups create predictable risks:

- Test workflows might accidentally delete production collections

- Developers might unintentionally modify indexes used by live services

- Widespread use of the `root` account makes actions impossible to trace or audit

- A compromised application might gain unrestricted access to all vector data

As usage grows, relying on informal conventions or shared admin accounts is no longer sustainable. A consistent, enforceable access model becomes essential—and this is exactly what Milvus RBAC provides.


## What is RBAC in Milvus

[RBAC (Role-Based Access Control)](https://milvus.io/docs/rbac.md) is a permission model that controls access based on **roles** rather than individual users. In Milvus, RBAC lets you define exactly which operations a user or service is allowed to perform—and on which specific resources. It provides a structured, scalable way to manage security as your system grows from a single developer to a complete production environment.

Milvus RBAC is built around the following core components:

![Users Roles Privileges](https://assets.zilliz.com/users_roles_privileges_030620f913.png)

- **Resource**: The entity being accessed. In Milvus, resources include the **instance**, **database**, and **collection**.

- **Privilege**: A specific allowed operation on a resource—for example, creating a collection, inserting data, or deleting entities. 

- **Privilege Group**: A predefined set of related privileges, such as “read-only” or “write.”

- **Role**: A combination of privileges and the resources they apply to. A role determines _what_ operations can be performed and _where_.

- **User**: An identity in Milvus. Each user has a unique ID and is assigned one or more roles.

These components form a clear hierarchy:

1. **Users are assigned roles**

2. **Roles define privileges**

3. **Privileges apply to specific resources**

A key design principle in Milvus is that **permissions are never assigned directly to users**. All access goes through roles. This indirection simplifies administration, reduces configuration errors, and makes permission changes predictable.

This model scales cleanly in real deployments. When multiple users share a role, updating the role’s privileges instantly updates permissions for all of them—without modifying each user individually. It’s a single point of control aligned with how modern infrastructure manages access.


## How RBAC Works in Milvus

When a client sends a request to Milvus, the system evaluates it through a series of authorization steps. Each step must pass before the operation is allowed to proceed:

![How RBAC Works in Milvus](https://assets.zilliz.com/how_rbac_works_afe48bc717.png)

1. **Authenticate the request:** Milvus first verifies the user identity. If authentication fails, the request is rejected with an authentication error. 

2. **Check role assignment:** After authentication, Milvus checks whether the user has at least one role assigned. If no role is found, the request is rejected with a permission denied error.

3. **Verify required privileges:** Milvus then evaluates whether the user’s role grants the required privilege on the target resource. If the privilege check fails, the request is rejected with a permission denied error.

4. **Execute the operation:** If all checks pass, Milvus executes the requested operation and returns the result.


## How to Configure Access Control via RBAC in Milvus

### 1. Prerequisites

Before RBAC rules can be evaluated and enforced, user authentication must be enabled so that every request to Milvus can be associated with a specific user identity.

Here are two standard deployment methods.

- **Deploying with Docker Compose**

If Milvus is deployed using Docker Compose, edit the `milvus.yaml` configuration file and enable authorization by setting `common.security.authorizationEnabled` to `true`:

```
common:
  security:
    authorizationEnabled: true
```

- **Deploying with Helm Charts**

If Milvus is deployed using Helm Charts, edit the `values.yaml` file and add the following configuration under `extraConfigFiles.user.yaml`:

```
extraConfigFiles:
  user.yaml: |+
    common:
      security:
        authorizationEnabled: true
```

### 2. Initialization

By default, Milvus creates a built-in `root` user when the system starts. The default password for this user is `Milvus`. 

As an initial security step, use the `root` user to connect to Milvus and change the default password immediately. It is strongly recommended to use a complex password to prevent unauthorized access.

```
from pymilvus import MilvusClient
# Connect to Milvus using the default root user
client = MilvusClient(
    uri='http://localhost:19530', 
    token="root:Milvus"
)
# Update the root password
client.update_password(
    user_name="root",
    old_password="Milvus", 
    new_password="xgOoLudt3Kc#Pq68"
)
```

### 3. Core Operations

**Create Users**

For daily usage, it is recommended to create dedicated users instead of using the `root` account. 


```
client.create_user(user_name="user_1", password="P@ssw0rd")
```

**Create Roles**

Milvus provides a built-in `admin` role with full administrative privileges. For most production scenarios, however, it is recommended to create custom roles to achieve finer-grained access control.


```
client.create_role(role_name="role_a")
```

**Create Privilege Groups**

A privilege group is a collection of multiple privileges. To simplify permission management, related privileges can be grouped and granted together.

Milvus includes the following built-in privilege groups:

- `COLL_RO`, `COLL_RW`, `COLL_ADMIN`

- `DB_RO`, `DB_RW`, `DB_ADMIN`

- `Cluster_RO`, `Cluster_RW`, `Cluster_ADMIN`

Using these built-in privilege groups can significantly reduce the complexity of permission design and improve consistency across roles.

You can either use the built-in privilege groups directly or create custom privilege groups as needed.

```
# Create a privilege group
client.create_privilege_group(group_name='privilege_group_1'）
# Add privileges to the privilege group
client.add_privileges_to_group(group_name='privilege_group_1', privileges=['Query', 'Search'])
```

**Grant Privileges or Privilege Groups to Roles**

After a role is created, privileges or privilege groups can be granted to the role. The target resources for these privileges can be specified at different levels, including the instance, database, or individual Collections.

```
client.grant_privilege_v2(
    role_name="role_a",
    privilege="Search",
    collection_name='collection_01',
    db_name='default',
)
client.grant_privilege_v2(
    role_name="role_a",
    privilege="privilege_group_1",
    collection_name='collection_01',
    db_name='default',
)
client.grant_privilege_v2(
    role_name="role_a",
    privilege="ClusterReadOnly",
    collection_name='*',
    db_name='*',
)
```

**Grant Roles to Users**

Once roles are assigned to a user, the user can access resources and perform the operations defined by those roles. A single user can be granted one or multiple roles, depending on the required access scope.

```
client.grant_role(user_name="user_1", role_name="role_a")
```

### 4. Inspect and Revoke Access

**Inspect Roles Assigned to a User**

```
client.describe_user(user_name="user_1")
```

**Inspect Privileges Assigned to a Role**

```
client.describe_role(role_name="role_a")
```

**Revoke Privileges from a Role**

```
client.revoke_privilege_v2(
    role_name="role_a",
    privilege="Search",
    collection_name='collection_01',
    db_name='default',
)
client.revoke_privilege_v2(
    role_name="role_a",
    privilege="privilege_group_1",
    collection_name='collection_01',
    db_name='default',
)
```

**Revoke Roles from a User**

```
client.revoke_role(
    user_name='user_1',
    role_name='role_a'
)
```

**Delete Users and Roles**

```
client.drop_user(user_name="user_1")
client.drop_role(role_name="role_a")
```

## Example: Access Control Design for a Milvus-Powered RAG System

Consider a Retrieval-Augmented Generation (RAG) system built on top of Milvus.

In this system, different components and users have clearly separated responsibilities, and each requires a different level of access.

| Actor                    | Responsibility                      | Required Access               |
| ------------------------ | ----------------------------------- | ----------------------------- |
| Platform Administrator   | System operations and configuration | Instance-level administration |
| Vector Ingestion Service | Vector data ingestion and updates   | Read and write access         |
| Search Service           | Vector search and retrieval         | Read-only access              |

```
from pymilvus import MilvusClient
client = MilvusClient(
    uri='http://localhost:19530',
    token="root:xxx"  # Replace with the updated root password
)
# 1. Create a user (use a strong password)
client.create_user(user_name="rag_admin", password="xxx")
client.create_user(user_name="rag_reader", password="xxx")
client.create_user(user_name="rag_writer", password="xxx")
# 2. Create roles
client.create_role(role_name="role_admin")
client.create_role(role_name="role_read_only")
client.create_role(role_name="role_read_write")
# 3. Grant privileges to the role
## Using built-in Milvus privilege groups
client.grant_privilege_v2(
    role_name="role_admin",
    privilege="Cluster_Admin",
    collection_name='*',
    db_name='*',
)
client.grant_privilege_v2(
    role_name="role_read_only",
    privilege="COLL_RO",
    collection_name='*',
    db_name='default',
)
client.grant_privilege_v2(
    role_name="role_read_write",
    privilege="COLL_RW",
    collection_name='*',
    db_name='default',
)
# 4. Assign the role to the user
client.grant_role(user_name="rag_admin", role_name="role_admin")
client.grant_role(user_name="rag_reader", role_name="role_read_only")
client.grant_role(user_name="rag_writer", role_name="role_read_write")
```

## Quick Tips: How to Operate Access Control Safely in Production

To ensure access control remains effective and manageable in long-running production systems, follow these practical guidelines.

**1. Change the default** `root` **password and limit the use of** `root` **account**

Update the default `root` password immediately after initialization and restrict its use to administrative tasks only. Avoid using or sharing the root account for routine operations. Instead, create dedicated users and roles for day-to-day access to reduce risk and improve accountability.

**2. Physically isolate Milvus instances across environments**

Deploy separate Milvus instances for development, staging, and production. Physical isolation provides a stronger safety boundary than logical access control alone and significantly reduces the risk of cross-environment mistakes.

**3. Follow the principle of least privilege**

Grant only the permissions required for each role:

- **Development environments:** permissions can be more permissive to support iteration and testing

- **Production environments:** permissions should be strictly limited to what is necessary

- **Regular audits:** periodically review existing permissions to ensure they are still required

**4. Actively revoke permissions when they are no longer needed**

Access control is not a one-time setup—it requires ongoing maintenance. Revoke roles and privileges promptly when users, services, or responsibilities change. This prevents unused permissions from accumulating over time and becoming hidden security risks.


## Conclusion

Configuring access control in Milvus is not inherently complex, but it is essential for operating the system safely and reliably in production. With a well-designed RBAC model, you can:

- **Reduce risk** by preventing accidental or destructive operations

- **Improve security** by enforcing least-privilege access to vector data

- **Standardize operations** through a clear separation of responsibilities

- **Scale with confidence**, laying the foundation for multi-tenant and large-scale deployments

Access control is not an optional feature or a one-time task. It is a foundational part of operating Milvus safely over the long term.

👉 Start building a solid security baseline with [RBAC](https://milvus.io/docs/rbac.md) for your Milvus deployment.

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Milvus RBAC Guide: How to Control Access to Your Vector Database


Milvus RBAC Explained: Secure Your Vector Database with Role-Based Access Control 



As unstructured data continues to explode and AI models keep getting smarter, vector search has become the default retrieval layer for many AI systems—RAG pipelines, AI search, agents, recommendation engines, and more. It works because it captures meaning: not just the words users type, but the intent behind them. 

Once these applications move into production, however, teams often discover that semantic understanding is only one side of the retrieval problem. Many workloads also depend on strict textual rules—such as matching exact terminology, preserving word order, or identifying phrases that carry technical, legal, or operational significance.

[Milvus 2.6](https://milvus.io/docs/release_notes.md#v267) removes that split by introducing native full-text search directly into the vector database. With token and positional indexes built into the core engine, Milvus can interpret a query’s semantic intent while enforcing precise keyword and phrase-level constraints. The result is a unified retrieval pipeline in which meaning and structure reinforce each other rather than living in separate systems.

[Phrase Match](https://milvus.io/docs/phrase-match.md) is a key part of this full-text capability. It identifies sequences of terms that appear together and in order—crucial for detecting log patterns, error signatures, product names, and any text in which word order defines meaning. In this post, we’ll explain how [Phrase Match](https://milvus.io/docs/phrase-match.md) works in [Milvus](https://milvus.io/), how `slop` adds flexibility needed for real-world text, and why these features make hybrid vector–full-text search not just possible but practical within a single database.


## What is Phrase Match? 

Phrase Match is a full-text query type in Milvus that focuses on _structure_—specifically, whether a sequence of words appears in the same order inside a document. When no flexibility is allowed, the query behaves strictly: the terms must appear next to each other and in sequence. A query like **"robotics machine learning"** therefore matches only when those three words occur as a continuous phrase. 

The challenge is that real text rarely behaves this neatly. Natural language introduces noise: extra adjectives slip in, logs reorder fields, product names gain modifiers, and human authors don’t write with query engines in mind. A strict phrase match breaks easily—one inserted word, one rephrasing, or one swapped term can cause a miss. And in many AI systems, especially production-facing ones, missing a relevant log line or rule-triggering phrase isn’t acceptable.

Milvus 2.6 addresses this friction with a simple mechanism: **slop**. Slop defines _the amount of wiggle room allowed between query_ terms. Instead of treating a phrase as brittle and inflexible, slop lets you decide whether one extra word is tolerable, or two, or even whether slight reordering should still count as a match. This moves phrase search from a binary pass–fail test to a controlled, tunable retrieval tool.

To see why this matters, imagine searching logs for all variants of the familiar networking error **"connection reset by peer."** In practice, your logs might look like:

```
connection reset by peer
connection fast reset by peer
connection was suddenly reset by the peer
peer reset connection by ...
peer unexpected connection reset happened
```

At a glance, all of these represent the same underlying event. But common retrieval methods struggle:


### BM25 struggles with structure.

It views the query as a bag of keywords, ignoring the order in which they appear. As long as “connection” and “peer” show up somewhere, BM25 may rank the document highly—even if the phrase is reversed or unrelated to the concept you're actually searching for.


### Vector search struggles with constraints.

Embeddings excel at capturing meaning and semantic relationships, but they cannot enforce a rule like “these words must appear in this sequence.” You might retrieve semantically related messages, but still miss the exact structural pattern required for debugging or compliance.

Phrase Match fills the gap between these two approaches. By using **slop**, you can specify exactly how much variation is acceptable: 

- `slop = 0` — Exact match (All terms must appear contiguously and in order.)

- `slop = 1` — Allow one extra word (Covers common natural-language variations with a single inserted term.)

- `slop = 2` — Allow multiple inserted words (Handles more descriptive or verbose phrasing.)

- `slop = 3` — Allow reordering (Supports reversed or loosely ordered phrases, often the hardest case in real-world text.)

Instead of hoping the scoring algorithm “gets it right,” you explicitly declare the structural tolerance your application requires.


## How Phrase Match Works in Milvus 

Powered by the [Tantivy](https://github.com/quickwit-oss/tantivy) search engine library, Phrase Match in Milvus is implemented on top of an inverted index with positional information. Instead of only checking whether terms appear in a document, it verifies that they appear in the right order and within a controllable distance. 

The diagram below illustrates the process:

![](https://assets.zilliz.com/phrase_match_workflow_a4f3badb66.png)

**1. Document Tokenization (with Positions)**

When documents are inserted into Milvus, text fields are processed by an [analyzer](https://milvus.io/docs/analyzer-overview.md), which splits the text into tokens (words or terms) and records each token’s position within the document. For example, `doc_1` is tokenized as: `machine (pos=0), learning (pos=1), boosts (pos=2), efficiency (pos=3)`. 

**2. Inverted Index Creation**

Next, Milvus builds an inverted index. Instead of mapping documents to their contents, the inverted index maps each token to the documents in which it appears, along with all recorded positions of that token within each document.

**3. Phrase Matching**

When a phrase query is executed, Milvus first uses the inverted index to identify documents that contain all query tokens. It then validates each candidate by comparing token positions to ensure the terms appear in the correct order and within the allowed `slop` distance. Only documents that satisfy both conditions are returned as matches.

The diagram below summarizes how Phrase Match works end-to-end.

![](https://assets.zilliz.com/workflow2_63c168b107.png)

## How to Enable Phrase Match in Milvus

Phrase Match works on fields of type **`VARCHAR`**, the string type in Milvus. To use it, you must configure your collection schema so that Milvus performs text analysis and stores positional information for the field. This is done by enabling two parameters: `enable_analyzer` and `enable_match`.


### Set enable_analyzer and enable_match

To turn on Phrase Match for a specific VARCHAR field, set both parameters to `True` when defining the field schema. Together, they tell Milvus to:

- **tokenize** the text (via `enable_analyzer`), and

- **build an inverted index with positional offsets** (via `enable_match`).

Phrase Match relies on both steps: the analyzer breaks text into tokens, and the match index stores where those tokens appear, enabling efficient phrase and slop-based queries.

Below is an example schema configuration that enables Phrase Match on a `text` field:

```
from pymilvus import MilvusClient, DataType

schema = MilvusClient.create_schema(enable_dynamic_field=False)
schema.add_field(
    field_name="id",
    datatype=DataType.INT64,
    is_primary=True,
    auto_id=True
)
schema.add_field(
    field_name='text',                 # Name of the field
    datatype=DataType.VARCHAR,         # Field data type set as VARCHAR (string)
    max_length=1000,                   # Maximum length of the string
    enable_analyzer=True,              # Enables text analysis (tokenization)
    enable_match=True                  # Enables inverted indexing for phrase matching
)
schema.add_field(
    field_name="embeddings",
    datatype=DataType.FLOAT_VECTOR,
    dim=5
)
```

## Search with Phrase Match: How Slop Affects the Candidate Set

Once you’ve enabled match for a VARCHAR field in your collection schema, you can perform phrase matches using the `PHRASE_MATCH` expression.

Note: The `PHRASE_MATCH` expression is case-insensitive. You can use either `PHRASE_MATCH` or `phrase_match`.

In search operations, Phrase Match is commonly applied before vector similarity ranking. It first filters documents based on explicit textual constraints, narrowing the candidate set. The remaining documents are then re-ranked using vector embeddings.

The example below shows how different `slop` values affect this process. By adjusting the `slop` parameter, you directly control which documents pass the phrase filter and proceed to the vector ranking stage.

Suppose you have a collection named `tech_articles` containing the following five entities:


| **doc_id** | **text**                                                          |
| ----------- | ----------------------------------------------------------------- |
| 1           | Machine learning boosts efficiency in large-scale data analysis   |
| 2           | Learning a machine-based approach is vital for modern AI progress |
| 3           | Deep learning machine architectures optimize computational loads  |
| 4           | Machine swiftly improves model performance for ongoing learning   |
| 5           | Learning advanced machine algorithms expands AI capabilities      |

**`slop=1`**

Here, we allow a slop of 1. The filter is applied to documents that contain the phrase “learning machine” with slight flexibility.

```
# Example: Filter documents containing "learning machine" with slop=1
filter_slop1 = "PHRASE_MATCH(text, 'learning machine', 1)"

result_slop1 = client.search(
    collection_name="tech_articles",
    anns_field="embeddings",
    data=[query_vector],
    filter=filter_slop1,
    search_params={"params": {"nprobe": 10}},
    limit=10,
    output_fields=["id", "text"]
)
```

Match results:

| doc_id | text                                                              |
| ------- | ----------------------------------------------------------------- |
| 2       | Learning a machine-based approach is vital for modern AI progress |
| 3       | Deep learning machine architectures optimize computational loads  |
| 5       | Learning advanced machine algorithms expands AI capabilities      |

**`slop=2`**

This example allows a slop of 2, meaning that up to two extra tokens (or reversed terms) are allowed between the words “machine” and “learning”.

```
# Example: Filter documents containing "machine learning" with slop=2
filter_slop2 = "PHRASE_MATCH(text, 'machine learning', 2)"

result_slop2 = client.search(
    collection_name="tech_articles",
    anns_field="embeddings",             # Vector field name
    data=[query_vector],                 # Query vector
    filter=filter_slop2,                 # Filter expression
    search_params={"params": {"nprobe": 10}},
    limit=10,                            # Maximum results to return
    output_fields=["id", "text"]
)
```

Match results:

| **doc_id** |                             **text**                             |
| :---------: | :--------------------------------------------------------------: |
|      1      |  Machine learning boosts efficiency in large-scale data analysis |
|      3      | Deep learning machine architectures optimize computational loads |

**`slop=3`**

In this example, a slop of 3 provides even more flexibility. The filter searches for “machine learning” with up to three token positions allowed between the words.

```
# Example: Filter documents containing "machine learning" with slop=3
filter_slop3 = "PHRASE_MATCH(text, 'machine learning', 3)"

result_slop2 = client.search(
    collection_name="tech_articles",
    anns_field="embeddings",             # Vector field name
    data=[query_vector],                 # Query vector
    filter=filter_slop3,                 # Filter expression
    search_params={"params": {"nprobe": 10}},
    limit=10,                            # Maximum results to return
    output_fields=["id", "text"]
)
```

Match results:

| **doc_id** |                              **text**                             |
| :---------: | :---------------------------------------------------------------: |
|      1      |  Machine learning boosts efficiency in large-scale data analysis  |
|      2      | Learning a machine-based approach is vital for modern AI progress |
|      3      |  Deep learning machine architectures optimize computational loads |
|      5      |    Learning advanced machine algorithms expands AI capabilities   |


## Quick Tips: What You Need to Know Before Enabling Phrase Match in Milvus

Phrase Match provides support for phrase-level filtering, but enabling it involves more than query-time configuration. It is helpful to be aware of the associated considerations before applying it in a production setting.

- Enabling Phrase Match on a field creates an inverted index, which increases storage usage. The exact cost depends on factors such as text length, the number of unique tokens, and the analyzer configuration. When working with large text fields or high-cardinality data, this overhead should be considered upfront.

- Analyzer configuration is another critical design choice. Once an analyzer is defined in the collection schema, it cannot be changed. Switching to a different analyzer later requires dropping the existing collection and recreating it with a new schema. For this reason, analyzer selection should be treated as a long-term decision rather than an experiment.

- Phrase Match behavior is tightly coupled to how text is tokenized. Before applying an analyzer to an entire collection, it is recommended to use the `run_analyzer` method to inspect the tokenization output and confirm that it matches your expectations. This step can help avoid subtle mismatches and unexpected query results later. For more information, refer to [Analyzer Overview](https://milvus.io/docs/analyzer-overview.md#share-DYZvdQ2vUowWEwx1MEHcdjNNnqT).


## Conclusion

Phrase Match is a core full-text search type that enables phrase-level and positional constraints beyond simple keyword matching. By operating on token order and proximity, it provides a predictable and precise way to filter documents based on how terms actually appear in text.

In modern retrieval systems, Phrase Match is commonly applied before vector-based ranking. It first restricts the candidate set to documents that explicitly satisfy required phrases or structures. Vector search is then used to rank these results by semantic relevance. This pattern is especially effective in scenarios such as log analysis, technical documentation search, and RAG pipelines, where textual constraints must be enforced before semantic similarity is considered.

With the introduction of the `slop` parameter in Milvus 2.6, Phrase Match becomes more tolerant of natural language variation while retaining its role as a full-text filtering mechanism. This makes phrase-level constraints easier to apply in production retrieval workflows.

👉 Try it out with the [demo](https://github.com/openvino-book/Milvus-Phrase-Match-Demo) scripts, and explore [Milvus 2.6](https://milvus.io/docs/release_notes.md#v267) to see how phrase-aware retrieval fits into your stack.

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Phrase Match with Slop: Better Full-Text Search Accuracy in Milvus


Phrase Match with Slop in Milvus 2.6: How to Improve Phrase-Level Full-Text Search Accuracy 


How to Safely Upgrade from Milvus 2.5.x to Milvus 2.6.x



[Retrieval Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation) has become the default approach for providing context and memory for AI applications — AI agents, customer support assistants, knowledge bases, and search systems all rely on it. 

In almost every RAG pipeline, the standard process is the same: take the documents, split them into chunks, and then embed those chunks for similarity retrieval in a vector database like [Milvus](https://milvus.io/). Because **chunking** happens upfront, the quality of those chunks directly affects how well the system retrieves information and how accurate the final answers are.

The issue is that traditional chunking strategies usually split text without any semantic understanding. Fixed-length chunking cuts based on token counts, and recursive chunking uses surface-level structure, but both still ignore the actual meaning of the text. As a result, related ideas often get separated, unrelated lines get grouped together, and important context gets fragmented.

In this blog, I’d like to share a different chunking strategy: [**Max–Min Semantic Chunking**](https://link.springer.com/article/10.1007/s10791-025-09638-7). Instead of chunking first, it embeds the text upfront and uses semantic similarity to decide where boundaries should form. By embedding before cutting, the pipeline can track natural shifts in meaning rather than relying on arbitrary length limits.

## How a Typical RAG Pipeline Works

Most RAG pipelines, regardless of the framework, follow the same four-stage assembly line. You’ve probably written some version of this yourself:


### 1. Data Cleaning and Chunking

The pipeline starts by cleaning the raw documents: removing headers, footers, navigation text, and anything that isn’t real content. Once the noise is out, the text gets split into smaller pieces. Most teams use fixed-size chunks — typically 300–800 tokens — because it keeps the embedding model manageable. The downside is that the splits are based on length, not meaning, so the boundaries can be arbitrary.


### 2. Embedding and Storage

Each chunk is then embedded using an embedding model like OpenAI’s [`text-embedding-3-small`](https://zilliz.com/ai-models/text-embedding-3-small) or BAAI’s encoder. The resulting vectors are stored in a vector database such as [Milvus](https://milvus.io/) or [Zilliz Cloud](https://zilliz.com/cloud). The database handles indexing and similarity search so you can quickly compare new queries against all stored chunks.


### 3. Querying

When a user asks a question — for example, _“How does RAG reduce hallucinations?”_ — the system embeds the query and sends it to the database. The database returns the top-K chunks whose vectors are closest to the query. These are the pieces of text the model will rely on to answer the question.


### 4. Answer Generation

The retrieved chunks are bundled together with the user query and fed into an LLM. The model generates an answer using the provided context as grounding.

**Chunking sits at the start of this whole pipeline, but it has an outsized impact**. If the chunks align with the natural meaning of the text, retrieval feels accurate and consistent. If the chunks were cut in awkward places, the system has a harder time finding the right information, even with strong embeddings and a fast vector database.


### The Challenges of Getting Chunking Right

Most RAG systems today use one of two basic chunking methods, both of which have limitations.

**1. Fixed-size chunking**

This is the simplest approach: split the text by a fixed token or character count. It’s fast and predictable, but completely unaware of grammar, topics, or transitions. Sentences can get cut in half. Sometimes even words. The embeddings you get from these chunks tend to be noisy because the boundaries don’t reflect how the text is actually structured.

**2. Recursive character splitting**

This method is a bit smarter. It splits text hierarchically based on cues like paragraphs, line breaks, or sentences. If a section is too long, it recursively divides it further. The output is generally more coherent, but still inconsistent. Some documents lack clear structure or have uneven section lengths, which hurts retrieval accuracy. And in some cases, this approach still produces chunks that exceed the model’s context window.

Both methods face the same tradeoff: precision vs. context. Smaller chunks improve retrieval accuracy but lose surrounding context; larger chunks preserve meaning but risk adding irrelevant noise. Striking the right balance is what makes chunking both foundational—and frustrating—in RAG system design.


## Max–Min Semantic Chunking: Embed First, Chunk Later

In 2025, S.R. Bhat et al. published [_Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Analysis_](https://arxiv.org/abs/2505.21700). One of their key findings was that there isn’t a single **“best”** chunk size for RAG. Small chunks (64–128 tokens) tend to work better for factual or lookup-style questions, while larger chunks (512–1024 tokens) help with narrative or high-level reasoning tasks. In other words, fixed-size chunking is always a compromise.

This raises a natural question: instead of picking one length and hoping for the best, can we chunk by meaning rather than size? [**Max–Min Semantic Chunking**](https://link.springer.com/article/10.1007/s10791-025-09638-7) is one approach I found that tries to do precisely that.

The idea is simple: **embed first, chunk second**. Instead of splitting text and then embedding whatever pieces fall out, the algorithm embeds _all sentences_ up front. It then uses the semantic relationships between those sentence embeddings to decide where the boundaries should go.

![Diagram showing embed-first chunk-second workflow in Max-Min Semantic Chunking](https://assets.zilliz.com/embed_first_chunk_second_94f69c664c.png)

Conceptually, the method treats chunking as a constrained clustering problem in embedding space. You walk through the document in order, one sentence at a time. For each sentence, the algorithm compares its embedding with those in the current chunk. If the new sentence is semantically close enough, it joins the chunk. If it’s too far, the algorithm starts a new chunk. The key constraint is that chunks must follow the original sentence order — no reordering, no global clustering.

The result is a set of variable-length chunks that reflect where the document’s meaning actually changes, instead of where a character counter happens to hit zero.


## How the Max–Min Semantic Chunking Strategy Works

Max–Min Semantic Chunking determines chunk boundaries by comparing how sentences relate to one another in the high-dimensional vector space. Instead of relying on fixed lengths, it looks at how meaning shifts across the document. The process can be broken down into six steps:


### 1. Embed all sentences and start a chunk

The embedding model converts each sentence in the document into a vector embedding. It processes sentences in order. If the first _n–k_ sentences form the current chunk C, the following sentence (sₙ₋ₖ₊₁) needs to be evaluated: should it join C, or start a new chunk?


### 2. Measure how consistent the current chunk is

Within chunk C, calculate the minimum pairwise cosine similarity among all sentence embeddings. This value reflects how closely related the sentences within the chunk are. A lower minimum similarity indicates that the sentences are less related, suggesting that the chunk may need to be split.


### 3. Compare the new sentence to the chunk

Next, calculate the maximum cosine similarity between the new sentence and any sentence already in C. This reflects how well the new sentence aligns semantically with the existing chunk.


### 4. Decide whether to extend the chunk or start a new one

This is the core rule:

- If the **new sentence’s max similarity** to chunk **C** is **greater than or equal to the** **minimum similarity inside C**, → The new sentence fits and stays in the chunk.

- Otherwise, → start a new chunk.

This ensures that each chunk maintains its internal semantic consistency.


### 5. Adjust thresholds as the document changes

To optimize chunk quality, parameters such as chunk size and similarity thresholds can be adjusted dynamically. This allows the algorithm to adapt to varying document structures and semantic densities.


### 6. Handle the first few sentences

When a chunk contains only one sentence, the algorithm handles the first comparison using a fixed similarity threshold. If the similarity between sentence 1 and sentence 2 is above that threshold, they form a chunk. If not, they split immediately.


## Strengths and Limitations of Max–Min Semantic Chunking

Max–Min Semantic Chunking improves how RAG systems split text by using meaning instead of length, but it’s not a silver bullet. Here’s a practical look at what it does well and where it still falls short.


### What It Does Well

Max–Min Semantic Chunking improves on traditional chunking in three important ways:


#### **1. Dynamic, meaning-driven chunk boundaries**

Unlike fixed-size or structure-based approaches, this method relies on semantic similarity to guide chunking. It compares the minimum similarity within the current chunk (how cohesive it is) to the maximum similarity between the new sentence and that chunk (how well it fits). If the latter is higher, the sentence joins the chunk; otherwise, a new chunk starts.


#### **2. Simple, practical parameter tuning**

The algorithm depends on just three core hyperparameters:

- the **maximum chunk size**,

- the **minimum similarity** between the first two sentences, and

- the **similarity threshold** for adding new sentences.

These parameters adjust automatically with context—larger chunks require stricter similarity thresholds to maintain coherence.


#### **3. Low processing overhead**

Because the RAG pipeline already computes sentence embeddings, Max–Min Semantic Chunking doesn’t add heavy computation. All it needs is a set of cosine similarity checks while scanning through sentences. This makes it cheaper than many semantic chunking techniques that require extra models or multi-stage clustering.


### What It Still Can’t Solve

Max–Min Semantic Chunking improves chunk boundaries, but it doesn’t eliminate all the challenges of document segmentation. Because the algorithm processes sentences in order and only clusters locally, it can still miss long-range relationships in longer or more complex documents.

One common issue is **context fragmentation**. When important information is spread across different parts of a document, the algorithm may place those pieces into separate chunks. Each chunk then carries only part of the meaning. 

For example, in the Milvus 2.4.13 Release Notes, as shown below, one chunk might contain the version identifier while another contains the feature list. A query like _“What new features were introduced in Milvus 2.4.13?”_ depends on both. If those details are split across different chunks, the embedding model may not connect them, leading to weaker retrieval.

+ ![Example showing context fragmentation in Milvus 2.4.13 Release Notes with version identifier and feature list in separate chunks](https://assets.zilliz.com/v2413_a98e1b1f99.png)

This fragmentation also affects the LLM generation stage. If the version reference is in one chunk and the feature descriptions are in another, the model receives incomplete context and can’t reason cleanly about the relationship between the two.

To mitigate these cases, systems often use techniques such as sliding windows, overlapping chunk boundaries, or multi-pass scans. These approaches reintroduce some of the missing context, reduce fragmentation, and help the retrieval step retain related information.


## Conclusion

Max–Min Semantic Chunking isn’t a magic fix for every RAG problem, but it does give us a more sane way to think about chunk boundaries. Instead of letting token limits decide where ideas get chopped, it uses embeddings to detect where the meaning actually shifts. For many real-world documents—APIs, specs, logs, release notes, troubleshooting guides—this alone can push retrieval quality noticeably higher.

What I like about this approach is that it fits naturally into existing RAG pipelines. If you already embed sentences or paragraphs, the extra cost is basically a few cosine similarity checks. You don’t need extra models, complex clustering, or heavyweight preprocessing. And when it works, the chunks it produces feel more “human”—closer to how we mentally group information when reading.

But the method still has blind spots. It only sees meaning locally, and it can’t reconnect information that’s intentionally spread apart. Overlapping windows, multi-pass scans, and other context-preserving tricks are still necessary, especially for documents where references and explanations live far from each other.

Still, Max–Min Semantic Chunking moves us in the right direction: away from arbitrary text slicing and toward retrieval pipelines that actually respect semantics. If you’re exploring ways to make RAG more reliable, it’s worth experimenting with.

Have questions or want to dig deeper into improving RAG performance? Join our [Discord](https://discord.com/invite/8uyFbECzPX) and connect with engineers who are building and tuning real retrieval systems every day.


Max–Min Semantic Chunking: Top Chunking Strategy to Improve RAG Performance


Embedding First, Then Chunking: Smarter RAG Retrieval with Max–Min Semantic Chunking 



If you’ve worked with long-running LLM conversations, you’ve probably had this frustrating moment: halfway through a long thread, the model starts drifting. Answers get vague, reasoning weakens, and key details mysteriously vanish. But if you drop the exact same prompt into a new chat, suddenly the model behaves—focused, accurate, grounded.

This isn’t the model “getting tired” — it’s **context rot**. As a conversation grows, the model has to juggle more information, and its ability to prioritize slowly declines. [Antropic studies](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) show that as context windows stretch from around 8K tokens to 128K, retrieval accuracy can drop by 15–30%. The model still has room, but it loses track of what matters. Bigger context windows help delay the problem, but they don’t eliminate it.

This is where **context engineering** comes in. Instead of handing the model everything at once, we shape what it sees: retrieving only the pieces that matter, compressing what no longer needs to be verbose, and keeping prompts and tools clean enough for the model to reason over. The goal is simple: make important information available at the right moment, and ignore the rest.

Retrieval plays a central role here, especially for long-running agents. Vector databases like [**Milvus**](https://milvus.io/) provide the foundation for efficiently pulling relevant knowledge back into context, letting the system stay grounded even as tasks grow in depth and complexity.

In this blog, we’ll look at how context rot happens, the strategies teams use to manage it, and the architectural patterns — from retrieval to prompt design — that keep AI agents sharp across long, multi-step workflows.


## Why Context Rot Happens 

People often assume that giving an AI model more context naturally leads to better answers. But that’s not actually true. Humans struggle with long inputs too: cognitive science suggests our working memory holds roughly **7±2 chunks** of information. Push beyond that, and we start to forget, blur, or misinterpret details.

LLMs show similar behavior—just at a much larger scale and with more dramatic failure modes.

The root issue comes from the [Transformer architecture](https://zilliz.com/learn/decoding-transformer-models-a-study-of-their-architecture-and-underlying-principles) itself. Every token must compare itself against every other token, forming pairwise attention across the entire sequence. That means computation grows **O(n²)** with context length. Expanding your prompt from 1K tokens to 100K doesn’t make the model “work harder”—it multiplies the number of token interactions by **10,000×**.

![](https://assets.zilliz.com/contextual_dilution_622033db72.png)

**Then there’s the problem with the training data.** Models see far more short sequences than long ones. So when you ask an LLM to operate across extremely large contexts, you’re pushing it into a regime it wasn’t heavily trained for. In practice, very long-context reasoning is often **out of distribution** for most models.

Despite these limits, long context is now unavoidable. Early LLM applications were mostly single-turn tasks—classification, summarization, or simple generation. Today, more than 70% of enterprise AI systems rely on agents that stay active across many rounds of interaction, often for hours, managing branching, multi-step workflows. Long-lived sessions have moved from exception to default.

Then the next question is: **how do we keep the model’s attention sharp without overwhelming it?** 


## Context Retrieval Approaches to Solving Context Rot

Retrieval is one of the most effective levers we have to combat context rot, and in practice it tends to show up in complementary patterns that address context rot from different angles.


### 1. Just-in-Time Retrieval: Reducing Unnecessary Context

One major cause of context rot is _overloading_ the model with information it doesn’t need yet. Claude Code—Anthropic’s coding assistant—solves this issue with **Just-in-Time (JIT) retrieval**, a strategy where the model fetches information only when it becomes relevant.

Instead of stuffing entire codebases or datasets into its context (which greatly increases the chance of drift and forgetting), Claude Code maintains a tiny index: file paths, commands, and documentation links. When the model needs a piece of information, it retrieves that specific item and inserts it into context **at the moment it matters**—not before. 

For example, if you ask Claude Code to analyze a 10GB database, it never tries to load the whole thing. It works more like an engineer:

1. Runs a SQL query to pull high-level summaries of the dataset.

2. Uses commands like `head` and `tail` to view sample data and understand its structure.

3. Retains only the most important information—such as key statistics or sample rows—within the context.

By minimizing what’s kept in context, JIT retrieval prevents the buildup of irrelevant tokens that cause rot. The model stays focused because it only ever sees the information required for the current reasoning step.


### 2. Pre-retrieval (Vector Search): Preventing Context Drift Before It Starts

Sometimes the model can’t “ask” for information dynamically—customer support, Q&A systems, and agent workflows often need the right knowledge available _before_ generation begins. This is where **pre-retrieval** becomes critical.

Context rot often happens because the model is handed a large pile of raw text and expected to sort out what matters. Pre-retrieval flips that: a vector database (like [Milvus](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/cloud)) identifies the most relevant pieces _before_ inference, ensuring only high-value context reaches the model.

In a typical RAG setup:

- Documents are embedded and stored in a vector database, such as Milvus.

- At query time, the system retrieves a small set of highly relevant chunks through similarity searches.

- Only those chunks go into the model’s context.

This prevents rot in two ways: 

- **Noise reduction:** irrelevant or weakly related text never enters the context in the first place.

- **Efficiency:** models process far fewer tokens, reducing the chance of losing track of essential details.

Milvus can search millions of documents in milliseconds, making this approach ideal for live systems where latency matters. 


### 3. Hybrid JIT and Vector Retrieval

Vector search-based pre-retrieval addresses a significant part of context rot by ensuring the model starts with high-signal information rather than raw, oversized text. But Anthropic highlights two real challenges that teams often overlook:

- **Timeliness:** If the knowledge base updates faster than the vector index is rebuilt, the model may rely on stale information.

- **Accuracy:** Before a task begins, it’s hard to predict precisely what the model will need—especially for multi-step or exploratory workflows.

So in real world workloads, a hybrid appaorch is the optimal solution. 

- Vector search for stable, high-confidence knowledge

- Agent-driven JIT exploration for information that evolves or only becomes relevant mid-task

By blending these two approaches, you get the speed and efficiency of vector retrieval for known information, and the flexibility for the model to discover and load new data whenever it becomes relevant. 

Let’s look at how this works in a real system. Take a production documentation assistant, for example. Most teams eventually settle on a two-stage pipeline: Milvus-powered vector search + agent-based JIT retrieval. 

**1. Milvus Powered Vector Search (Pre-retrieval)** 

- Convert your documentation, API references, changelogs, and known issues into embeddings.

- Store them in the Milvus Vector Database with metadata like product area, version, and update time.

- When a user asks a question, run a semantic search to grab the top-K relevant segments.

This resolves roughly 80% of routine queries in under 500 ms, giving the model a strong, context-rot-resistant starting point.

**2. Agent-Based Exploration**

When the initial retrieval isn’t sufficient—e.g., when the user asks for something highly specific or time-sensitive—the agent can call tools to fetch new information:

- Use `search_code` to locate specific functions or files in the codebase

- Use `run_query` to pull real-time data from the database

- Use `fetch_api` to obtain the latest system status

These calls typically take **3–5 seconds**, but they ensure the model always works with fresh, accurate, and relevant data—even for questions the system couldn’t anticipate beforehand.

This hybrid structure ensures context remains timely, correct, and task-specific, dramatically reducing the risk of context rot in long-running agent workflows.

Milvus is especially effective in these hybrid scenarios because it supports:

- **Vector search + scalar filtering**, combining semantic relevance with structured constraints

- **Incremental updates**, allowing embeddings to be refreshed without downtime

This makes Milvus an ideal backbone for systems that need both semantic understanding and precise control over what gets retrieved.

![](https://assets.zilliz.com/milvus_in_hybrid_architecture_7d4e391aa4.png)

For example, you might run a query like:
```
# You can combine queries like this in Milvus
collection.search(
    data=[query_embedding],  # Semantic similarity
    anns_field="embedding",
    param={"metric_type": "COSINE", "params": {"nprobe": 10}},
    expr="doc_type == 'API' and update_time > '2025-01-01'",  # Structured filtering
    limit=5
)
```

## How to Choose the Right Approach for Dealing With Context Rot

With vector-search pre-retrieval, Just-in-Time retrieval, and hybrid retrieval all available, the natural question is: **which one should you use?**

Here is a simple but practical way to choose—based on how _stable_ your knowledge is and how _predictable_ the model’s information needs are.


### 1. Vector Search → Best for Stable Domains

If the domain changes slowly but demands precision—finance, legal work, compliance, medical documentation—then a Milvus-powered knowledge base with **pre-retrieval** is usually the right fit.

The information is well-defined, updates are infrequent, and most questions can be answered by retrieving semantically relevant documents upfront.

**Predictable tasks + stable knowledge → Pre-retrieval.**


### 2. Just-in-Time Retrieval → Best for Dynamic, Exploratory Workflows

Fields like software engineering, debugging, analytics, and data science involve rapidly changing environments: new files, new data, new deployment states. The model can’t predict what it will need before the task starts. 

**Unpredictable tasks + fast-changing knowledge → Just-in-Time retrieval.**


### 3. Hybrid Approach → When Both Conditions Are True

Many real systems aren’t purely stable or purely dynamic. For example, developer documentation changes slowly, whereas the state of a production environment changes minute by minute. A hybrid approach lets you:

- Load known, stable knowledge using vector search (fast, low-latency)

- Fetch dynamic information with agent tools on demand (accurate, up-to-date)

**Mixed knowledge + mixed task structure → Hybrid retrieval approach.** 


## What if the Context Window Still Isn’t Enough

Context engineering helps reduce overload, but sometimes the problem is more fundamental: **the task simply won’t fit**, even with careful trimming. 

Certain workflows—like migrating a large codebase, reviewing multi-repository architectures, or generating deep research reports—can exceed 200K+ context windows before the model reaches the end of the task. Even with vector search doing heavy lifting, some tasks require more persistent, structured memory.

Recently, Anthropic has offered three practical strategies. 


### 1. Compression: Preserve Signal, Drop Noise

When the context window approaches its limit, the model can **compress earlier interactions** into concise summaries. Good compression keeps 

- Key decisions

- Constraints and requirements

- Outstanding issues

- Relevant samples or examples

And removes:

- Verbose tool outputs

- Irrelevant logs

- Redundant steps

The challenge is balance. Compress too aggressively, and the model loses critical information; compress too lightly, and you gain little space. Effective compression keeps the “why” and “what” while discarding the “how we got here.”


### 2. Structured Note-Taking: Move Stable Information Outside Context

Instead of keeping everything inside the model’s window, the system can store important facts in **external memory**—a separate database or a structured store that the agent can query as needed.

For example, Claude’s Pokémon-agent prototype stores durable facts like:

- `Pikachu leveled up to 8`

- `Trained 1234 steps on Route 1`

- `Goal: reach level 10`

Meanwhile, transient details—battle logs, long tool outputs—stay outside the active context. This mirrors how humans use notebooks: we don’t keep every detail in our working memory; we store reference points externally and look them up when needed.

Structured note-taking prevents context rot caused by repeated, unnecessary details while giving the model a reliable source of truth.


### 3. Sub-Agent Architecture: Divide and Conquer Large Tasks

For complex tasks, a multi-agent architecture can be designed where a lead agent oversees the overall work, while several specialized sub-agents handle specific aspects of the task. These sub-agents dive deep into large amounts of data related to their sub-tasks but only return the concise, essential results. This approach is commonly used in scenarios like research reports or data analysis.

![](https://assets.zilliz.com/longduration_task_cbbc07b9ca.png)

In practice, it's best to start by using a single agent combined with compression to handle the task. External storage should only be introduced when there’s a need to retain memory across sessions. The multi-agent architecture should be reserved for tasks that genuinely require parallel processing of complex, specialized sub-tasks.

Each approach extends the system's effective “working memory” without blowing the context window—and without triggering context rot.


## Best Practices for Designing Context That Actually Works

After handling context overflow, there’s another equally important piece: how the context is built in the first place. Even with compression, external notes, and sub-agents, the system will struggle if the prompt and tools themselves aren’t designed to support long, complex reasoning.

![](https://assets.zilliz.com/System_Prompts_cf655dcd0d.png)

Anthropic offers a helpful way to think about this—less as a single prompt-writing exercise, and more as constructing context across three layers.

### **System Prompts: Find the Goldilocks Zone**

Most system prompts fail at the extremes. Too much detail—lists of rules, nested conditions, hard-coded exceptions—makes the prompt brittle and difficult to maintain. Too little structure leaves the model guessing what to do.

The best prompts sit in the middle: structured enough to guide behavior, flexible enough for the model to reason. In practice, this means giving the model a clear role, a general workflow, and light tool guidance—nothing more, nothing less.

For example:

```
You are a technical documentation assistant serving developers.
1. Start by retrieving relevant documents from the Milvus knowledge base.  
2. If the retrieval results are insufficient, use the `search_code` tool to perform a deeper search in the codebase.  
3. When answering, cite specific documentation sections or code line numbers.

## Tool guidance
- search_docs: Used for semantic retrieval, best for conceptual questions.  
- search_code: Used for precise lookup in the codebase, best for implementation-detail questions.  
…
```

This prompt sets direction without overwhelming the model or forcing it to juggle dynamic information that doesn’t belong here.


### Tool Design: Less Is More

Once the system prompt sets the high-level behavior, tools carry the actual operational logic. A surprisingly common failure mode in tool-augmented systems is simply having too many tools—or having tools whose purposes overlap.

A good rule of thumb:

- **One tool, one purpose**

- **Explicit, unambiguous parameters**

- **No overlapping responsibilities**

If a human engineer would hesitate about which tool to use, the model will too. Clean tool design reduces ambiguity, lowers cognitive load, and prevents context from being cluttered with unnecessary tool attempts.

![](https://assets.zilliz.com/tooling_complexity_7d2bb60c54.png)

### Dynamic Information Should Be Retrieved, Not Hardcoded

The final layer is the easiest to overlook. Dynamic or time-sensitive information—such as status values, recent updates, or user-specific state—should not appear in the system prompt at all. Baking it into the prompt guarantees it will become stale, bloated, or contradictory over long tasks.

Instead, this information should be fetched only when needed, either through retrieval or via agent tools. Keeping dynamic content out of the system prompt prevents context rot and keeps the model’s reasoning space clean.


## Conclusion

As AI agents move into production environments across different industries, they’re taking on longer workflows and more complex tasks than ever before. In these settings, managing context becomes a practical necessity.

**However, a bigger context window doesn’t automatically produce better results**; in many cases, it does the opposite. When a model is overloaded, fed stale information, or forced through massive prompts, accuracy quietly drifts. That slow, subtle decline is what we now call **context rot**. 

Techniques like JIT retrieval, pre-retrieval, hybrid pipelines, and vector-database-powered semantic search all aim at the same goal: **making sure the model sees the right information at the right moment — no more, no less — so it can stay grounded and produce reliable answers.**

As an open-source, high-performance vector database, [**Milvus**](https://milvus.io/) sits at the core of this workflow. It provides the infrastructure for storing knowledge efficiently and retrieving the most relevant pieces with low latency. Paired with JIT retrieval and other complementary strategies, Milvus helps AI agents remain accurate as their tasks become deeper and more dynamic. 

But retrieval is only one piece of the puzzle. Good prompt design, a clean and minimal toolset, and sensible overflow strategies — whether compression, structured notes, or sub-agents — all work together to keep the model focused across long-running sessions. This is what real context engineering looks like: not clever hacks, but thoughtful architecture.

If you want AI agents that stay accurate over hours, days, or entire workflows, context deserves the same attention you’d give to any other core part of your stack. 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Context Engineering Strategies to Prevent LLM Context Rot with Milvus


Keeping AI Agents Grounded: Context Engineering Strategies that Prevent Context Rot Using Milvus 


How to Build AI Search Pipelines That Understand User Intent with Milvus


Building AI Search Pipelines That Understand What Users Want with Milvus 


LangChain 1.0 and Milvus: Build Production-Ready AI Agents with Long-Term Memory


LangChain 1.0 and Milvus: How to Build Production-Ready Agents with Real Long-Term Memory


Milvus Ngram Index: Faster Keyword Matching and LIKE Queries for Agent Workloads


Introducing the Milvus Ngram Index: Faster Keyword Matching and LIKE Queries for Agent Workloads


Milvus Tiered Storage: 80% Less Vector Search Cost with On-Demand Hot–Cold Data Loading


Stop Paying for Cold Data: 80% Cost Reduction with On-Demand Hot–Cold Data Loading in Milvus Tiered Storage



As AI systems move from experiments to production infrastructure, vector databases are no longer dealing with millions of embeddings. **Billions are now routine, and tens of billions are increasingly common.** At this scale, algorithmic choices affect not only performance and recall, but also translate directly into infrastructure cost.

This leads to a core question for large-scale deployments: **how do you choose the right index to deliver acceptable recall and latency without letting compute resource usage spiral out of control?**

Graph-based indexes such as **NSW, HNSW, CAGRA, and Vamana** have become the most widely adopted answer. By navigating pre-built neighborhood graphs, these indexes enable fast nearest-neighbor search at billion scale, avoiding brute-force scanning and comparison of every vector against the query.

However, the cost profile of this approach is uneven. **Querying a graph is relatively cheap; constructing it is not.** Building a high-quality graph requires large-scale distance computations and iterative refinement across the entire dataset—workloads that traditional CPU resources struggle to handle efficiently as data grows.

NVIDIA’s CAGRA addresses this bottleneck by using GPUs to accelerate graph construction through massive parallelism. While this significantly reduces build time, relying on GPUs for both index construction and query serving introduces higher cost and scalability constraints in production environments.

To balance these tradeoffs, [Milvus 2.6.1](https://milvus.io/docs/release_notes.md#v261) **adopts a hybrid design for** [GPU_CAGRA](https://milvus.io/docs/gpu-cagra.md) **indexes**: **GPUs are used only for graph construction, while query execution runs on CPUs.** This preserves the quality advantages of GPU-built graphs while keeping query serving scalable and cost-efficient—making it especially well suited for workloads with infrequent data updates, large query volumes, and strict cost sensitivity.


## What Is CAGRA and How Does It Work?

Graph-based vector indexes generally fall into two major categories:

- **Iterative graph construction**, represented by **CAGRA** (already supported in Milvus).

- **Insert-based graph construction**, represented by **Vamana** (currently under development in Milvus).

These two approaches differ significantly in their design goals and technical foundations, making each suitable for different data scales and workload patterns.

**NVIDIA CAGRA (CUDA ANN Graph-based)** is a GPU-native algorithm for approximate nearest neighbor (ANN) search, designed for building and querying large-scale proximity graphs efficiently. By leveraging GPU parallelism, CAGRA significantly accelerates graph construction and delivers high-throughput query performance compared with CPU-based approaches such as HNSW.

CAGRA is built on the **NN-Descent (Nearest Neighbor Descent)** algorithm, which constructs a k-nearest-neighbor (kNN) graph through iterative refinement. In each iteration, candidate neighbors are evaluated and updated, gradually converging toward higher-quality neighborhood relationships across the dataset.

After each refinement round, CAGRA applies additional graph pruning techniques—such as **2-hop detour pruning**—to remove redundant edges while preserving search quality. This combination of iterative refinement and pruning results in a **compact yet well-connected graph** that is efficient to traverse at query time.

Through repeated refinement and pruning, CAGRA produces a graph structure that supports **high recall and low-latency nearest-neighbor search at large scale**, making it particularly well suited for static or infrequently updated datasets.


### Step 1: Building the Initial Graph with NN-Descent

NN-Descent is based on a simple but powerful observation: if node _u_ is a neighbor of _v_, and node _w_ is a neighbor of _u_, then _w_ is very likely a neighbor of _v_ as well. This transitive property allows the algorithm to discover true nearest neighbors efficiently, without exhaustively comparing every pair of vectors.

CAGRA uses NN-Descent as its core graph construction algorithm. The process works as follows:

**1. Random initialization:** Each node starts with a small set of randomly selected neighbors, forming a rough initial graph.

**2. Neighbor expansion:** In each iteration, a node gathers its current neighbors and their neighbors to form a candidate list. The algorithm computes similarities between the node and all candidates. Because each node’s candidate list is independent, these computations can be assigned to separate GPU thread blocks and executed in parallel at a massive scale.

**3. Candidate list update:** If the algorithm finds candidates that are closer than the node’s current neighbors, it swaps out the more distant neighbors and updates the node’s kNN list. Over multiple iterations, this process produces a much higher-quality approximate kNN graph.

**4. Convergence check:** As iterations progress, fewer neighbor updates occur. Once the number of updated connections drops below a set threshold, the algorithm stops, indicating the graph has effectively stabilized.

Because neighbor expansion and similarity computation for different nodes are fully independent, CAGRA maps each node’s NN-Descent workload to a dedicated GPU thread block. This design enables massive parallelism and makes graph construction orders of magnitude faster than traditional CPU-based methods.


### Step 2: Pruning the Graph with 2-Hop Detours

After NN-Descent completes, the resulting graph is accurate but overly dense. NN-Descent intentionally keeps extra candidate neighbors, and the random initialization phase introduces many weak or irrelevant edges. As a result, each node often ends up with a degree two times—or even several times—higher than the target degree.

To produce a compact and efficient graph, CAGRA applies 2-hop detour pruning.

The idea is straightforward: if node _A_ can reach node _B_ indirectly through a shared neighbor _C_ (forming a path A → C → B), and the distance of this indirect path is comparable to the direct distance between _A_ and _B_, then the direct edge A → B is considered redundant and can be removed.

![](https://assets.zilliz.com/2_hop_detours_d15eae8702.png)

A key advantage of this pruning strategy is that each edge’s redundancy check depends only on local information—the distances between the two endpoints and their shared neighbors. Because every edge can be evaluated independently, the pruning step is highly parallelizable and fits naturally onto GPU batch execution. 

As a result, CAGRA can prune the graph efficiently on GPUs, reducing storage overhead by **40–50%** while preserving search accuracy and improving traversal speed during query execution.


## GPU_CAGRA in Milvus: What’s Different?

While GPUs offer major performance advantages for graph construction, production environments face a practical challenge: GPU resources are far more expensive and limited than CPUs. If both index building and query execution depend solely on GPUs, several operational issues quickly emerge:

- **Low resource utilization:** Query traffic is often irregular and bursty, leaving GPUs idle for long periods and wasting expensive compute capacity.

- **High deployment cost:** Assigning a GPU to every query-serving instance drives up hardware costs, even though most queries do not fully utilize GPU performance.

- **Limited scalability:** The number of GPUs available directly caps how many service replicas you can run, restricting your ability to scale with demand.

- **Reduced flexibility:** When both index building and querying depend on GPUs, the system becomes tied to GPU availability and cannot easily shift workloads to CPUs.

To address these constraints, Milvus 2.6.1 introduces a flexible deployment mode for the GPU_CAGRA index through the `adapt_for_cpu` parameter. This mode enables a hybrid workflow: CAGRA uses the GPU to build a high-quality graph index, while query execution runs on the CPU—typically using HNSW as the search algorithm. 

In this setup, GPUs are used where they deliver the most value—fast, high-accuracy index construction—while CPUs handle large-scale query workloads in a far more cost-effective and scalable manner.

As a result, this hybrid approach is particularly well suited for workloads where:

- **Data updates are infrequent**, so index rebuilds are rare

- **Query volume is high**, requiring many inexpensive replicas

- **Cost sensitivity is high**, and GPU usage must be tightly controlled


### Understanding `adapt_for_cpu`

In Milvus, the `adapt_for_cpu` parameter controls how a CAGRA index is serialized to disk during index building and how it is deserialized into memory at load time. By changing this setting at build time and load time, Milvus can flexibly switch between GPU-based index construction and CPU-based query execution.

Different combinations of `adapt_for_cpu` at build time and load time result in four execution modes, each designed for a specific operational scenario.



| **Build Time (`adapt_for_cpu`)** | **Load Time (`adapt_for_cpu`)** |                                  **Execution Logic**                                 |                              **Recommended Scenario**                              |
| :------------------------------: | :-----------------------------: | :----------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------: |
|             **true**             |             **true**            |  Build with GPU_CAGRA → serialize as HNSW → deserialize as HNSW → **CPU querying**  |                 Cost-sensitive workloads; large-scale query serving                |
|             **true**             |            **false**            |  Build with GPU_CAGRA → serialize as HNSW → deserialize as HNSW → **CPU querying**  |        Ensuing queries fall back to the CPU when parameter mismatches occur        |
|             **false**            |             **true**            |  Build with GPU_CAGRA → serialize as CAGRA → deserialize as HNSW → **CPU querying** | Keeping the original CAGRA index for storage while enabling a temporary CPU search |
|             **false**            |            **false**            | Build with GPU_CAGRA → serialize as CAGRA → deserialize as CAGRA → **GPU querying** |               Performance-critical workloads where cost is secondary               |

**Note:** The `adapt_for_cpu` mechanism supports only one-way conversion. A CAGRA index can be converted into HNSW because the CAGRA graph structure preserves all neighbor relationships that HNSW needs. However, an HNSW index cannot be converted back to CAGRA, as it lacks the additional structural information needed for GPU-based querying. As a result, the build-time settings should be selected carefully, with consideration for long-term deployment and querying requirements.


## Putting GPU_CAGRA to the Test

To evaluate the effectiveness of the hybrid execution model—using GPUs for index construction and CPUs for query execution—we conducted a series of controlled experiments in a standardized environment. The evaluation focuses on three dimensions: **index build performance**, **query performance**, and **recall accuracy**.

**Experimental Setup**

The experiments were performed on widely adopted, industry-standard hardware to ensure the results remain reliable and broadly applicable.

- CPU: MD EPYC 7R13 Processor(16 cpus)

- GPU: NVIDIA L4


### 1. Index Build Performance

We compare CAGRA built on the GPU with HNSW built on the CPU, under the same target graph degree of 64.

![](https://assets.zilliz.com/cp1_a177200ab2.png)

**Key Findings**

- **GPU CAGRA builds indexes 12–15× faster than CPU HNSW.** On both Cohere1M and Gist1M, GPU-based CAGRA significantly outperforms CPU-based HNSW, highlighting the efficiency of GPU parallelism during graph construction.

- **Build time increases linearly with NN-Descent iterations.** As iteration counts rise, build time grows in a near-linear manner, reflecting the iterative refinement nature of NN-Descent and providing a predictable trade-off between build cost and graph quality.


### 2. Query performance

In this experiment, the CAGRA graph is built once on the GPU and then queried using two different execution paths:

- **CPU querying**: the index is deserialized into HNSW format and searched on the CPU

- **GPU querying**: search runs directly on the CAGRA graph using GPU-based traversal

![](https://assets.zilliz.com/cp2_bd00e60553.png)

**Key Findings**

- **GPU search throughput is 5–6× higher than CPU search.** Across both Cohere1M and Gist1M, GPU-based traversal delivers substantially higher QPS, highlighting the efficiency of parallel graph navigation on GPUs.

- **Recall increases with NN-Descent iterations, then plateaus.** As the number of build iterations grows, recall improves for both CPU and GPU querying. However, beyond a certain point, additional iterations yield diminishing gains, indicating that graph quality has largely converged.


### 3. Recall accuracy

In this experiment, both CAGRA and HNSW are queried on the CPU to compare recall under identical query conditions.

![](https://assets.zilliz.com/cp3_1a46a7bdda.png)

**Key Findings**

**CAGRA achieves higher recall than HNSW on both datasets**, showing that even when a CAGRA index is built on the GPU and deserialized for CPU search, the graph quality is well preserved.


## What’s Next: Scaling Index Construction with Vamana

Milvus’s hybrid GPU–CPU approach offers a practical and cost-efficient solution for today’s large-scale vector search workloads. By building high-quality CAGRA graphs on GPUs and serving queries on CPUs, it combines fast index construction with scalable, affordable query execution—**particularly well suited for workloads with infrequent updates, high query volumes, and strict cost constraints.**

At even larger scales—**tens or hundreds of billions of vectors**—index construction itself becomes the bottleneck. When the full dataset no longer fits into GPU memory, the industry typically turns to **insert-based graph construction** methods such as **Vamana**. Instead of building the graph all at once, Vamana processes data in batches, incrementally inserting new vectors while maintaining global connectivity.

Its construction pipeline follows three key stages:

**1. Geometric batch growth** — beginning with small batches to form a skeleton graph, then increasing batch size to maximize parallelism, and finally using large batches to refine details.

**2. Greedy insertion** — each new node is inserted by navigating from a central entry point, iteratively refining its neighbor set.

**3. Backward edge updates** — adding reverse connections to preserve symmetry and ensure efficient graph navigation.

Pruning is integrated directly into the construction process using the α-RNG criterion: if a candidate neighbor _v_ is already covered by an existing neighbor _p′_ (i.e., _d(p′, v) < α × d(p, v)_), then _v_ is pruned. The parameter α allows precise control over sparsity and accuracy. GPU acceleration is achieved through in-batch parallelism and geometric batch scaling, striking a balance between index quality and throughput.

Together, these techniques enable teams to handle rapid data growth and large-scale index updates without running into GPU memory limitations.

![](https://assets.zilliz.com/One_more_thing_b458360e25.png)

The Milvus team is actively building Vamana support, with a targeted release in the first half of 2026. Stay tuned. 

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


## Learn More about Milvus 2.6 Features

- [Introducing Milvus 2.6: Affordable Vector Search at Billion Scale](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)

- [Introducing the Embedding Function: How Milvus 2.6 Streamlines Vectorization and Semantic Search](https://milvus.io/blog/data-in-and-data-out-in-milvus-2-6.md)

- [JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility](https://milvus.io/blog/json-shredding-in-milvus-faster-json-filtering-with-flexibility.md)

- [Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus](https://milvus.io/blog/unlocking-true-entity-level-retrieval-new-array-of-structs-and-max-sim-capabilities-in-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)



Optimizing CAGRA in Milvus: A Hybrid GPU–CPU Approach


Optimizing NVIDIA CAGRA in Milvus: A Hybrid GPU–CPU Approach to Faster Indexing and Cheaper Queries


AISAQ in Milvus Cuts Memory 3,200× for Billion-Scale Search


Introducing AISAQ in Milvus: Billion-Scale Vector Search Just Got 3,200× Cheaper on Memory



As AI systems are increasingly applied to real-time decision-making, geospatial data becomes increasingly important in a growing set of applications—particularly those that operate in the physical world or serve users across real locations. 

Consider food delivery platforms like DoorDash or Uber Eats. When a user places an order, the system isn’t simply calculating the shortest distance between two points. It evaluates restaurant quality, courier availability, live traffic conditions, service areas, and increasingly, user and item embeddings that represent personal preferences. Similarly, autonomous vehicles must perform path planning, obstacle detection, and scene-level semantic understanding under strict latency constraints—often within milliseconds. In these domains, effective decisions depend on combining spatial constraints with semantic similarity, rather than treating them as independent steps.

At the data layer, however, spatial and semantic data have traditionally been handled by separate systems.

- Geospatial databases and spatial extensions are designed to store coordinates, polygons, and spatial relationships such as containment or distance. 

- Vector databases handle vector embeddings that represent the data’s semantic meaning. 

When applications need both, they are often forced into multi-stage query pipelines—filtering by location in one system, then performing vector search in another. This separation increases system complexity, adds query latency, and makes it difficult to perform spatial–semantic reasoning efficiently at scale.

[Milvus 2.6](https://milvus.io/docs/release_notes.md#v264) addresses this problem by introducing the [Geometry Field](https://milvus.io/docs/geometry-field.md), which allows vector similarity search to be combined directly with spatial constraints. This enables use cases such as:

- Location-Base Service (LBS): “find similar POIs within this city block”

- Multi‑modal search: “retrieve similar photos within 1km of this point”

- Maps & logistics: “assets inside a region” or “routes intersecting a path”

Paired with the new [RTREE index](https://milvus.io/docs/rtree.md)—a tree-based structure optimized for spatial filtering—Milvus now supports efficient geospatial operators like `st_contains`, `st_within`, and `st_dwithin` alongside high-dimensional vector search. Together, they make spatially aware intelligent retrieval not just possible, but practical.

In this post, we’ll walk through how the Geometry Field and RTREE index work, and how they combine with vector similarity search to enable real-world, spatial-semantic applications.


## What Is a Geometry Field in Milvus?

A **Geometry field** is a schema-defined data type (`DataType.GEOMETRY`) in Milvus used to store geometric data. Unlike systems that handle only raw coordinates, Milvus supports a range of spatial structures—including **Point**, **LineString**, and **Polygon**. 

This makes it possible to represent real-world concepts such as restaurant locations (Point), delivery zones (Polygon), or autonomous-vehicle trajectories (LineString), all within the same database that stores semantic vectors. In other words, Milvus becomes a unified system for both _where_ something is and _what it means_.

Geometry values are stored using the [Well-Known Text (WKT)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) format, a human-readable standard for inserting and querying geometric data. This simplifies data ingestion and querying because WKT strings can be inserted directly into a Milvus record. For example:

```
data = [
    { 
        "id": 1,
        "geo": "POINT(116.4074 39.9042)",
        "vector": vector,
    }
]
```

## What Is the RTREE Index and How Does It Work?

Once Milvus introduces the Geometry data type, it also needs an efficient way to filter spatial objects. Milvus handles this using a two-stage spatial filtering pipeline:

- **Coarse filtering:** Quickly narrows down candidates using spatial indexes such as RTREE.

- **Fine filtering:** Applies exact geometry checks on the candidates that remain, ensuring correctness at boundaries.

This design balances performance and accuracy. The spatial index aggressively prunes irrelevant data, while precise geometric checks ensure correct results for operators such as containment, intersection, and distance thresholds.

At the core of this pipeline is **RTREE (Rectangle Tree)**, a spatial indexing structure designed to accelerate queries over geometric data. RTREE works by organizing objects hierarchically using **Minimum Bounding Rectangles (MBRs)**, allowing large portions of the search space to be skipped during query execution.

### Phase 1: Building the RTREE Index

RTREE construction follows a bottom-up process that groups nearby spatial objects into increasingly larger bounding regions:

**1. Create leaf nodes:** For each geometry object, calculate its **Minimum Bounding Rectangle (MBR)**—the smallest rectangle that fully contains the object—and store it as a leaf node.

**2. Group into larger boxs:** Cluster nearby leaf nodes and wrap each group inside a new MBR, producing internal nodes.

**3. Add the root node:** Create a root node whose MBR covers all internal groups, forming a height-balanced tree structure.

![](https://assets.zilliz.com/RTREE_Index_11b5d09e07.png)

**Phase 2: Accelerate queries**

**1. Form the query MBR:** Calculate the MBR for the geometry used in your query.

**2. Prune branches:** Starting from the root, compare the query MBR with each internal node. Skip any branch whose MBR does not intersect with the query MBR.

**3. Collect candidates:** Descend into intersecting branches and gather the candidate leaf nodes.

**4. Perform exact matching:** For each candidate, run the spatial predicate to get precise results.


### Why RTREE Is Fast

RTREE delivers strong performance in spatial filtering because of several key design features:

- **Every node stores an MBR:** Each node approximates the area of all geometries in its subtree. This makes it easy to decide whether a branch should be explored during a query.

- **Fast pruning:** Only subtrees whose MBR intersects the query region are explored. Irrelevant areas are ignored entirely.

- **Scales with data size:** RTREE supports spatial searches in **O(log N)** time, enabling fast queries even as the dataset expands.

- **Boost.Geometry implementation:** Milvus builds its RTREE index using [Boost.Geometry](https://www.boost.org/library/latest/geometry/), a widely used C++ library that provides optimized geometry algorithms and a thread-safe RTREE implementation suitable for concurrent workloads.


### Supported geometry operators

Milvus provides a set of spatial operators that allow you to filter and retrieve entities based on geometric relationships. These operators are essential for workloads that need to understand how objects relate to one another in space.

The following table lists the [geometry operators](https://milvus.io/docs/geometry-operators.md) currently available in Milvus.



|       **Operator**       |                                                  **Description**                                                 |
| :----------------------: | :--------------------------------------------------------------------------------------------------------------: |
| **st_intersects(A, B)** |                        Returns TRUE if geometries A and B share at least one common point.                       |
|  **st_contains(A, B)**  |                Returns TRUE if geometry A completely contains geometry B (excluding the boundary).               |
|   **st_within(A, B)**   | Returns TRUE if geometry A is completely contained within geometry B. This is the inverse of st_contains(A, B). |
|   **st_covers(A, B)**   |                      Returns TRUE if geometry A covers geometry B (including the boundary).                      |
|   **st_touches(A, B)**  |           Returns TRUE if geometries A and B touch at their boundaries but do not intersect internally.          |
|   **st_equals(A, B)**   |                            Returns TRUE if geometries A and B are spatially identical.                           |
|  **st_overlaps(A, B)**  |            Returns TRUE if geometries A and B partially overlap and neither fully contains the other.            |
| **st_dwithin(A, B, d)** |                          Returns TRUE if the distance between A and B is less than _d_.                          |


### How to Combine Geolocation Index and Vector Index

With Geometry support and the RTREE index, Milvus can combine geospatial filtering with vector similarity search in a single workflow. The process works in two steps:

**1. Filter by location using RTREE:** Milvus first uses the RTREE index to narrow the search to entities within the specified geographic range (e.g., “within 2 km”).

**2. Rank by semantics using vector search:** From the remaining candidates, the vector index selects the Top-N most similar results based on embedding similarity.

![](https://assets.zilliz.com/Geometry_R_Tree_f1d88fc252.png)

## Real-World Use Cases of Geo-Vector Retrieval

### 1. Delivery Services: Smarter, Location-Aware Recommendations

Platforms such as DoorDash or Uber Eats handle hundreds of millions of requests each day. The moment a user opens the app, the system must determine—based on the user’s location, time of day, taste preferences, estimated delivery times, real-time traffic, and courier availability—which restaurants or couriers are the best match _right now_.

Traditionally, this requires querying a geospatial database and a separate recommendation engine, followed by multiple rounds of filtering and re-ranking. With the Geolocation Index, Milvus greatly simplifies this workflow:

- **Unified storage** — Restaurant coordinates, courier locations, and user preference embeddings all live in one system.

- **Joint retrieval** — First apply a spatial filter (e.g., _restaurants within 3 km_), then use vector search to rank by similarity, taste preference, or quality.

- **Dynamic decision-making** — Combine real-time courier distribution and traffic signals to quickly assign the nearest, most suitable courier.

This unified approach allows the platform to perform spatial and semantic reasoning in a single query. For example, when a user searches “curry rice,” Milvus retrieves restaurants that are semantically relevant _and_ prioritizes those that are nearby, deliver quickly, and match the user’s historical taste profile.


### 2. Autonomous Driving: More Intelligent Decisions

In autonomous driving, geospatial indexing is fundamental to perception, localization, and decision-making. Vehicles must continuously align themselves to high-definition maps, detect obstacles, and plan safe trajectories—all within just a few milliseconds.

With Milvus, the Geometry type and RTREE index can store and query rich spatial structures such as:

- **Road boundaries** (LineString)

- **Traffic regulation zones** (Polygon)

- **Detected obstacles** (Point)

These structures can be indexed efficiently, allowing geospatial data to take part directly in the AI decision loop. For example, an autonomous vehicle can quickly determine whether its current coordinates fall within a specific lane or intersect with a restricted area, simply through an RTREE spatial predicate.

When combined with vector embeddings generated by the perception system—such as scene embeddings that capture the current driving environment—Milvus can support more advanced queries, like retrieving historical driving scenarios similar to the current one within a 50-meter radius. This helps models interpret the environment faster and make better decisions.


## Conclusion

Geolocation is more than latitude and longitude. In location-sensitive applications, it provides essential context about **where events occur, how entities relate spatially, and how those relationships shape system behavior**. When combined with semantic signals from machine learning models, geospatial data enables a richer class of queries that are difficult to express—or inefficient to execute—when spatial and vector data are handled separately.

With the introduction of the Geometry Field and the RTREE index, Milvus brings vector similarity search and spatial filtering into a single query engine. This allows applications to perform joint retrieval across **vectors, geospatial data, and time**, supporting use cases such as spatially aware recommendation systems, multimodal location-based search, and region- or path-constrained analytics. More importantly, it reduces architectural complexity by eliminating multi-stage pipelines that move data between specialized systems.

As AI systems continue to move closer to real-world decision-making, understanding **_what_** content is relevant will increasingly need to be paired with **_where_** it applies and **_when_** it matters. Milvus provides the building blocks for this class of spatial-semantic workloads in a way that is both expressive and practical for operating at scale.

For more information about the Geometry Field and the RTREE index, check the documentation below: 

- [Geometry Field | Milvus Documentation](https://milvus.io/docs/geometry-field.md)

- [RTREE | Milvus Documentation](https://milvus.io/docs/rtree.md)

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


## Learn More about Milvus 2.6 Features

- [Introducing Milvus 2.6: Affordable Vector Search at Billion Scale](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)

- [Introducing the Embedding Function: How Milvus 2.6 Streamlines Vectorization and Semantic Search](https://milvus.io/blog/data-in-and-data-out-in-milvus-2-6.md)

- [JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility](https://milvus.io/blog/json-shredding-in-milvus-faster-json-filtering-with-flexibility.md)

- [Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus](https://milvus.io/blog/unlocking-true-entity-level-retrieval-new-array-of-structs-and-max-sim-capabilities-in-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus ](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)

- [Vector Search in the Real World: How to Filter Efficiently Without Killing Recall](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md)

Geospatial Filtering + Vector Search in Milvus with Geometry Fields and RTREE


Bringing Geospatial Filtering and Vector Search Together with Geometry Fields and RTREE in Milvus 2.6


Array of Structs in Milvus: Entity-Level Multi-Vector Retrieval


Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus



Modern AI systems are producing more semi-structured JSON data than ever before. Customer and product information are compacted to a JSON object, microservices emit JSON logs on every request, IoT devices stream sensor readings in lightweight JSON payloads, and today’s AI applications increasingly standardize on JSON for structured output. The result is a flood of JSON-like data flowing into vector databases. 

Traditionally, there are two ways to handle JSON documents:

- **Predefine every field of JSON into a fixed schema and build an index:** This approach delivers solid query performance, but it’s rigid. Once the data format changes, every new or modified field triggers another round of painful Data Definition Language (DDL) updates and schema migrations.

- **Store the entire JSON object as a single column (both JSON type and Dynamic Schema in Milvus use this approach):** This option offers excellent flexibility, but at the cost of query performance. Each request requires runtime JSON parsing and often a full table scan, resulting in latency that spikes as the dataset grows.

It used to be a dilemma of flexibility and performance. 

Not anymore with the newly introduced JSON Shredding feature in [Milvus](https://milvus.io/). 

With the introduction of [JSON Shredding](https://milvus.io/docs/json-shredding.md), Milvus now achieves schema-free agility with the performance of columnar storage, finally making large-scale semi-structured data both flexible and query-friendly.

## How JSON Shredding Works

JSON shredding speeds up JSON queries by transforming row-based JSON documents into highly optimized columnar storage. Milvus preserves the flexibility of JSON for data modeling while automatically optimizing columnar storage—significantly improving data access and query performance.

To handle sparse or rare JSON fields efficiently, Milvus also has an inverted index for shared keys. All of this happens transparently to users: you can insert JSON documents as usual, and leave it to Milvus to manage the optimal storage and indexing strategy internally.

When Milvus receives raw JSON records with varying shapes and structures, it analyzes each JSON key for its occurrence ratio and type stability (whether its data type is consistent across documents). Based on this analysis, each key is classified into one of three categories:

- **Typed keys:** Keys that appear in most documents and always have the same data type (e.g., all integers or all strings).

- **Dynamic keys**: Keys that appear frequently but have mixed data types (e.g., sometimes a string, sometimes an integer).

- **Shared keys:** Keys that are infrequent, sparse, or nested, falling below a configurable frequency threshold.

Milvus handles each category differently to maximize efficiency:

- **Typed keys** are stored in dedicated, strongly typed columns.

- **Dynamic keys** are placed into dynamic columns based on the actual value type observed at runtime.

- Both typed and dynamic columns are stored in Arrow/Parquet columnar formats for fast scanning and highly optimized query execution.

- **Shared keys** are consolidated into a compact binary-JSON column, accompanied by a shared-key inverted index. This index accelerates queries on low-frequency fields by pruning irrelevant rows early and restricting the search to only those documents that contain the queried key.

This combination of adaptive columnar storage and inverted indexing forms the core of Milvus’s JSON shredding mechanism, enabling both flexibility and high performance at scale.

The overall workflow is illustrated below:

![](https://assets.zilliz.com/json_shredding_79a62a9661.PNG)

Now that we’ve covered the basics of how JSON Shredding works, let’s take a closer look at the key capabilities that make this approach both flexible and high-performance.

### Shredding and Columnarization

When a new JSON document is written, Milvus breaks it down and reorganizes it into optimized columnar storage:

- Typed and dynamic keys are automatically identified and stored in dedicated columns.

- If the JSON contains nested objects, Milvus generates path-based column names automatically. For example, a `name` field inside a `user` object can be stored with the column name `/user/name`.

- Shared keys are stored together in a single, compact binary JSON column. Because these keys appear infrequently, Milvus builds an inverted index for them, enabling fast filtering and allowing the system to quickly locate the rows that contain the specified key.

### Intelligent Column Management

Beyond shredding JSON into columns, Milvus adds an additional layer of intelligence through dynamic column management, ensuring that JSON Shredding stays flexible as data evolves.

- **Columns created as needed:** When new keys appear in incoming JSON documents, Milvus automatically groups values with the same key into a dedicated column. This preserves the performance advantages of columnar storage without requiring users to design schemas upfront. Milvus also infers the data type of new fields (e.g., INTEGER, DOUBLE, VARCHAR) and selects an efficient columnar format for them.

- **Every key is handled automatically:** Milvus analyzes and processes every key in the JSON document. This ensures broad query coverage without forcing users to predefine fields or build indexes in advance. 

### Query Optimization

Once the data is reorganized into the right columns, Milvus selects the most efficient execution path for each query:

- **Direct column scans for typed and dynamic keys:** If a query targets a field that has already been split into its own column, Milvus can scan that column directly. This reduces the total amount of data that needs to be processed and leverages SIMD-accelerated columnar computation for even faster execution.

- **Indexed lookup for shared keys:** If the query involves a field that was not promoted into its own column—typically a rare key—Milvus evaluates it against the shared-key column. The inverted index built on this column allows Milvus to quickly identify which rows contain the specified key and skip over the rest, significantly improving performance for low-frequency fields.

- **Automatic metadata management:** Milvus continuously maintains global metadata and dictionaries so that queries remain accurate and efficient, even as the structure of incoming JSON documents evolves over time.

## Performance benchmarks

We designed a benchmark to compare the query performance of storing the entire JSON document as a single raw field versus using the newly released JSON Shredding feature.

### Test environment and methodology

- Hardware: 1 core/8GB cluster

- Dataset: 1 million documents from [JSONBench](https://github.com/ClickHouse/JSONBench.git)

- Methodology: Measure QPS and latency across different query patterns

### Results: typed keys

This test measured performance when querying a key present in most documents.

| Query Expression          | QPS (without shredding) | QPS (with shredding) | Performance Boost |
| ------------------------- | ----------------------- | -------------------- | ----------------- |
| json['time_us'] > 0     | 8.69                    | 287.5                | **33x**           |
| json['kind'] == 'commit' | 8.42                    | 126.1                | **14.9x**         |

### Results: shared keys

This test focused on querying sparse, nested keys that fall into the “shared” category.

| Query Expression                     | QPS (without shredding) | QPS (with shredding) | Performance Boost |
| ------------------------------------ | ----------------------- | -------------------- | ----------------- |
| json['identity']['seq'] > 0        | 4.33                    | 385                  | **88.9x**         |
| json['identity']['did'] == 'xxxxx' | 7.6                     | 352                  | **46.3x**         |

Shared-key queries show the most dramatic improvements (up to 89× faster), while typed-key queries deliver consistent 15–30× speedups. Overall, every query type benefits from JSON Shredding, with clear performance gains across the board.

## Try It Now

Whether you’re working with API logs, IoT sensor data, or rapidly evolving application payloads, JSON Shredding gives you the rare ability to have both flexibility and high performance. 

The feature is now available and welcome to try it out now. You can also check [this doc](https://milvus.io/docs/json-shredding.md) for more details.

Have questions or want a deep dive on any feature of the latest Milvus? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Milvus JSON Shredding: Faster JSON Filtering With Flexibility


JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility




If you’ve ever built a vector search application, you already know the workflow a little too well. Before any data can be stored, it must first be transformed into vectors using an embedding model, cleaned and formatted, and then finally ingested into your vector database. Every query goes through the same process as well: embed the input, run a similarity search, then map the resulting IDs back to your original documents or records. It works — but it creates a distributed tangle of preprocessing scripts, embedding pipelines, and glue code that you have to maintain.

[Milvus](https://milvus.io/), a high-performance open-source vector database, now takes a major step toward simplifying all of that. [Milvus 2.6](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md) introduces the **Data-in, Data-out feature (also known as the** [**Embedding Function**](https://milvus.io/docs/embedding-function-overview.md#Embedding-Function-Overview)**)**, a built-in embedding capability that connects directly to major model providers such as OpenAI, AWS Bedrock, Google Vertex AI, and Hugging Face. Instead of managing your own embedding infrastructure, Milvus can now call these models for you. You can also insert and query using raw text — and soon other data types — while Milvus automatically handles vectorization at write and query time. 

In the rest of this post, we’ll take a closer look at how Data-in, Data-out works under the hood, how to configure providers and embedding functions, and how you can use it to streamline your vector search workflows end-to-end. 


## What is Data-in, Data-out?

Data-in, Data-out in Milvus 2.6 is built on the new Function module — a framework that enables Milvus to handle data transformation and embedding generation internally, without any external preprocessing services. (You can follow the design proposal in [GitHub issue #35856](https://github.com/milvus-io/milvus/issues/35856).) With this module, Milvus can take raw input data, call an embedding provider directly, and automatically write the resulting vectors into your collection.

At a high level, the **Function** module turns embedding generation into a native database capability. Instead of running separate embedding pipelines, background workers, or reranker services, Milvus now sends requests to your configured provider, retrieves embeddings, and stores them alongside your data — all inside the ingestion path. This removes the operational overhead of managing your own embedding infrastructure.

Data-in, Data-out introduces three major improvements to the Milvus workflow:

- **Insert raw data directly** – You can now insert unprocessed text, images, or other data types directly into Milvus. No need to convert them into vectors in advance.

- **Configure one embedding function** – Once you configure an embedding model in Milvus, it automatically manages the entire embedding process. Milvus integrates seamlessly with a range of model providers, including OpenAI, AWS Bedrock, Google Vertex AI, Cohere, and Hugging Face.

- **Query with raw inputs** – You can now perform semantic search using raw text or other content-based queries. Milvus uses the same configured model to generate embeddings on the fly, perform similarity search, and return relevant results.

In short, Milvus now automatically embeds — and optionally reranks — your data. Vectorization becomes a built-in database function, eliminating the need for external embedding services or custom preprocessing logic.


## How Data-in, Data-out Works

The diagram below illustrates how Data-in, Data-out operates inside Milvus. 

![](https://assets.zilliz.com/diagram_data_in_data_out_4c9e06c884.png)

The Data-in, Data-out workflow can be broken down into six main steps:

1. **Input Data** – The user inserts raw data — such as text, images, or other content types — directly into Milvus without performing any external preprocessing.

2. **Generate Embeddings** – The Function module automatically invokes the configured embedding model through its third-party API, converting the raw input into vector embeddings in real time.

3. **Store Embeddings** – Milvus writes the generated embeddings into the designated vector field within your collection, where they become available for similarity search operations.

4. **Submit a Query** – The user issues a raw-text or content-based query to Milvus, just as with the input stage.

5. **Semantic Search** – Milvus embeds the query using the same configured model, runs a similarity search over the stored vectors, and determines the closest semantic matches.

6. **Return Results** – Milvus returns the top-k most similar results — mapped back to their original data — directly to the application.


## How to Configure Data-in, Data-out

### Prerequisites

- Install the latest version of **Milvus 2.6**.

- Prepare your embedding API key from a supported provider (e.g., OpenAI, AWS Bedrock, or Cohere). In this example, we’ll use **Cohere** as the embedding provider.


### Modify the `milvus.yaml` Configuration

If you are running Milvus with **Docker Compose**, you’ll need to modify the `milvus.yaml` file to enable the Function module. You can refer to the official documentation for guidance: [Configure Milvus with Docker Compose](https://milvus.io/docs/configure-docker.md?tab=component#Download-a-configuration-file) (Instructions for other deployment methods can also be found here).

In the configuration file, locate the sections `credential` and `function`.

Then, update the fields `apikey1.apikey` and `providers.cohere`.

```
...
credential:
  aksk1:
    access_key_id:  # Your access_key_id
    secret_access_key:  # Your secret_access_key
  apikey1:
    apikey: "***********************" # Edit this section
  gcp1:
    credential_json:  # base64 based gcp credential data
# Any configuration related to functions
function:
  textEmbedding:
    providers:
                        ...
      cohere: # Edit the section below
        credential:  apikey1 # The name in the crendential configuration item
        enable: true # Whether to enable cohere model service
        url:  "https://api.cohere.com/v2/embed" # Your cohere embedding url, Default is the official embedding url
      ...
...
```



Once you’ve made these changes, restart Milvus to apply the updated configuration.


## How to Use the Data-in, Data-out Feature

### 1. Define the Schema for the Collection 

To enable the embedding feature, your **collection schema** must include at least three fields:

- **Primary key field (**`id`**)** – Uniquely identifies each entity in the collection.

- **Scalar field (**`document`**)** – Stores the original raw data.

- **Vector field (**`dense`**)** – Stores the generated vector embeddings.

```
from pymilvus import MilvusClient, DataType, Function, FunctionType
# Initialize Milvus client
client = MilvusClient(
    uri="http://localhost:19530",
)
# Create a new schema for the collection
schema = client.create_schema()
# Add primary field "id"
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=False)
# Add scalar field "document" for storing textual data
schema.add_field("document", DataType.VARCHAR, max_length=9000)
# Add vector field "dense" for storing embeddings.
# IMPORTANT: Set `dim` to match the exact output dimension of the embedding model.
# For instance, OpenAI's text-embedding-3-small model outputs 1536-dimensional vectors.
# For dense vector, data type can be FLOAT_VECTOR or INT8_VECTOR
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=1536) # Set dim according to the embedding model you use.
```


### 2. Define the Embedding Function

Next, define the **embedding function** in the schema. 

- `name` – A unique identifier for the function.

- `function_type` – Set to `FunctionType.TEXTEMBEDDING` for text embeddings. Milvus also supports other function types such as `FunctionType.BM25` and `FunctionType.RERANK`. See [Full Text Search](https://milvus.io/docs/full-text-search.md#Full-Text-Search) and [Decay Ranker Overview](https://milvus.io/docs/decay-ranker-overview.md#Decay-Ranker-Overview) for more details.

- `input_field_names` – Defines the input field for raw data (`document`).

- `output_field_names` – Defines the output field where the vector embeddings will be stored (`dense`).

- `params` – Contains configuration parameters for the embedding function. The values for `provider` and `model_name` must match the corresponding entries in your `milvus.yaml` configuration file.

**Note:** Each function must have a unique `name` and `output_field_names` to distinguish different transformation logics and prevent conflicts.



```
# Define embedding function (example: OpenAI provider)
text_embedding_function = Function(
    name="cohere_embedding",                  # Unique identifier for this embedding function
    function_type=FunctionType.TEXTEMBEDDING, # Type of embedding function
    input_field_names=["document"],           # Scalar field to embed
    output_field_names=["dense"],             # Vector field to store embeddings
    params={                                  # Provider-specific configuration (highest priority)
        "provider": "cohere",                 # Embedding model provider
        "model_name": "embed-v4.0",     # Embedding model
        # "credential": "apikey1",            # Optional: Credential label
        # Optional parameters:
        # "dim": "1536",       # Optionally shorten the vector dimension
        # "user": "user123"    # Optional: identifier for API tracking
    }
)
# Add the embedding function to your schema
schema.add_function(text_embedding_function)
```




### 3. Configure the Index

Once the fields and functions are defined, create an index for the collection. For simplicity, we use the AUTOINDEX type here as an example.



```
# Prepare index parameters
index_params = client.prepare_index_params()
# Add AUTOINDEX to automatically select optimal indexing method
index_params.add_index(
    field_name="dense",
    index_type="AUTOINDEX",
    metric_type="COSINE" 
)
```




### 4. Create the Collection

Use the defined schema and index to create a new collection. In this example, we’ll create a collection named Demo.



```
# Create collection named "demo"
client.create_collection(
    collection_name='demo', 
    schema=schema, 
    index_params=index_params
)
```




### 5. Insert Data

Now you can insert raw data directly into Milvus — there’s no need to generate embeddings manually. 



```
# Insert sample documents
client.insert('demo', [
    {'id': 1, 'document': 'Milvus simplifies semantic search through embeddings.'},
    {'id': 2, 'document': 'Vector embeddings convert text into searchable numeric data.'},
    {'id': 3, 'document': 'Semantic search helps users find relevant information quickly.'},
])
```




### 6. Perform Vector Search

After inserting data, you can perform searches directly using raw text queries. Milvus automatically converts your query into an embedding, perform similarity search against stored vectors, and return the top matches.



```
# Perform semantic search
results = client.search(
    collection_name='demo', 
    data=['How does Milvus handle semantic search?'], # Use text query rather than query vector
    anns_field='dense',   # Use the vector field that stores embeddings
    limit=1,
    output_fields=['document'],
)
print(results)
# Example output:
# data: ["[{'id': 1, 'distance': 0.8821347951889038, 'entity': {'document': 'Milvus simplifies semantic search through embeddings.'}}]"]
```



For more details on vector search, see: [Basic Vector Search ](https://milvus.io/docs/single-vector-search.md)and [Query API](https://milvus.io/docs/get-and-scalar-query.md).


## Get Started with Milvus 2.6 

With Data-in, Data-out, Milvus 2.6 takes vector search simplicity to the next level. By integrating embedding and reranking functions directly within Milvus, you no longer need to manage external preprocessing or maintain separate embedding services.

Ready to try it out? Install [Milvus](https://milvus.io/docs) 2.6 today and experience the power of Data-in, Data-out for yourself. 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


## Learn More about Milvus 2.6 Features

- [Introducing Milvus 2.6: Affordable Vector Search at Billion Scale](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)

- [JSON Shredding in Milvus: 88.9x Faster JSON Filtering with Flexibility](https://milvus.io/blog/json-shredding-in-milvus-faster-json-filtering-with-flexibility.md)

- [Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in Milvus](https://milvus.io/blog/unlocking-true-entity-level-retrieval-new-array-of-structs-and-max-sim-capabilities-in-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus ](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)

- [Vector Search in the Real World: How to Filter Efficiently Without Killing Recall ](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md)


Introducing the Embedding Function: How Milvus 2.6 Streamlines Vectorization and Semantic Search


7 Years, 2 Major Rebuilds, 40K+ GitHub Stars: The Rise of Milvus as the Leading Open-Source Vector Database


Choosing a vector database for ANN search at Reddit


RAG-Anything and Milvus for Multimodal RAG Systems

Multimodal RAG Made Simple: RAG-Anything + Milvus Instead of 20 Separate Tools



Most developers start their agentic systems with a single agent and only later realize they’ve basically built a very expensive chatbot. For simple tasks, a ReAct-style agent works fine, but it quickly hits limits: it can’t run steps in parallel, it loses track of long reasoning chains, and it tends to fall apart once you add too many tools to the mix. Multi-agent setups promise to fix this, but they bring their own problems: coordination overhead, brittle handoffs, and a ballooning shared context that quietly erodes model quality.

[OpenAgents](https://github.com/OpenAgentsInc) is an open-source framework for building multi-agent systems in which AI agents work together, share resources, and tackle long-horizon projects within persistent communities. Instead of a single central orchestrator, OpenAgents lets agents collaborate in a more distributed way: they can discover each other, communicate, and coordinate around shared goals.

Paired with the [Milvus](https://milvus.io/) vector database, this pipeline gains a scalable, high-performance long-term memory layer. Milvus powers agent memory with fast semantic search, flexible indexing choices like HNSW and IVF, and clean isolation through partitioning, so agents can store, retrieve, and reuse knowledge without drowning in context or stepping on each other’s data.

In this post, we’ll walk through how OpenAgents enables distributed multi-agent collaboration, why Milvus is a critical foundation for scalable agent memory, and how to assemble such a system step by step.


## Challenges in Building Real-World Agent Systems

Many mainstream agent frameworks today—LangChain, AutoGen, CrewAI, and others—are built around a **task-centric** model. You spin up a set of agents, give them a job, maybe define a workflow, and let them run. This works well for narrow or short-lived use cases, but in real production environments, it exposes three structural limitations:

- **Knowledge remains siloed.** An agent’s experience is confined to its own deployment. A code-review agent in engineering doesn’t share what it learns with a product-team agent evaluating feasibility. Every team ends up rebuilding knowledge from scratch, which is both inefficient and brittle. 

- **Collaboration is rigid.** Even in multi-agent frameworks, cooperation usually depends on workflows defined in advance. When collaboration needs to shift, these static rules cannot adapt, making the entire system less flexible.

- **A lack of a persistent state.** Most agents follow a simple lifecycle: _start → execute → shut down._ They forget everything between runs—context, relationships, decisions made, and interaction history. Without a persistent state, agents cannot build long-term memory or evolve their behavior.

These structural issues come from treating agents as isolated task executors rather than participants in a broader collaborative network. 

The OpenAgents team believes that future agent systems need more than stronger reasoning—they need a mechanism that enables agents to discover one another, build relationships, share knowledge, and work together dynamically. And critically, this should not depend on a single central controller. The internet works because it’s distributed—no single node dictates everything, and the system becomes more robust and scalable as it grows. Multi-agent systems benefit from the same design principle. That’s why OpenAgents removes the idea of an all-powerful orchestrator and instead enables decentralized, network-driven cooperation.


## What’s OpenAgents? 

OpenAgents is an open-source framework for building AI agent networks that enables open collaboration, where AI agents work together, share resources, and tackle long-horizon projects. It provides the infrastructure for an internet of agents — where agents collaborate openly with millions of other agents in persistent, growing communities. At the technical level, the system is structured around three core components: **Agent Network, Network Mods, and Transports.**


### 1. Agent Network: A Shared Environment for Collaboration

An agent network is a shared environment where multiple agents can connect, communicate, and work together to solve complex tasks. Its core characteristics include:

- **Persistent operation:** Once created, the Network stays online independently of any single task or workflow.

- **Dynamic agent:** Agents can join at any time using a Network ID; no pre-registration required.

- **Multi-protocol support:** A unified abstraction layer supports communication over WebSocket, gRPC, HTTP, and libp2p.

- **Autonomous configuration:** Each Network maintains its own permissions, governance, and resources.

With just one line of code, you can spin up a Network, and any agent can join immediately through standard interfaces.


### 2. Network Mods: Pluggable Extensions for Collaboration

Mods provide a modular layer of collaboration features that stay decoupled from the core system. You can mix and match Mods based on your specific needs, enabling collaboration patterns tailored to each use case.


| **Mod**                 | **Purpose**                     | **Use cases**                                   |
| ----------------------- | ------------------------------- | ----------------------------------------------- |
| **Workspace Messaging** | Real-time message communication | Streaming responses, instant feedback           |
| **Forum**               | Asynchronous discussion         | Proposal reviews, multi-round deliberation      |
| **Wiki**                | Shared knowledge base           | Knowledge consolidation, document collaboration |
| **Social**              | Relationship graph              | Expert routing, trust networks                  |

All Mods operate on a unified event system, making it easy to extend the framework or introduce custom behaviors whenever required.


### 3. Transports: A Protocol-Agnostic Channel for Communication

Transports are the communication protocols that allow heterogeneous agents to connect and exchange messages within an OpenAgents network. OpenAgents supports multiple transport protocols that can run simultaneously inside the same network, including:

- **HTTP/REST** for broad, cross-language integration

- **WebSocket** for low-latency, bidirectional communication

- **gRPC** for high-performance RPC suited to large-scale clusters

- **libp2p** for decentralized, peer-to-peer networking

- **A2A**, an emerging protocol designed specifically for agent-to-agent communication

All transports operate through a unified event-based message format, enabling seamless translation between protocols. You don’t need to worry about which protocol a peer agent uses—the framework handles it automatically. Agents built in any language or framework can join an OpenAgents network without rewriting existing code.

## Integrating OpenAgents with Milvus for Long-Term Agentic Memory

OpenAgents solves the challenge of how agents **communicate, discover each other, and collaborate**—but collaboration alone isn’t enough. Agents generate insights, decisions, conversation history, tool results, and domain-specific knowledge. Without a persistent memory layer, all of that evaporates the moment an agent shuts down.

This is where **Milvus** becomes essential. Milvus provides the high-performance vector storage and semantic retrieval needed to turn agent interactions into durable, reusable memory. When integrated into the OpenAgents network, it offers three major advantages:


#### **1. Semantic Search**

Milvus delivers fast semantic search using indexing algorithms like HNSW and IVF_FLAT. Agents can retrieve the most relevant historical records based on meaning rather than keywords, enabling them to:

- recall prior decisions or plans,

- avoid repeating work,

- maintain long-horizon context across sessions.

This is the backbone of _agentic memory_: fast, relevant, contextual retrieval.


#### **2. Billion-Scale Horizontal Scalability**

Real agent networks generate massive amounts of data. Milvus is built to operate comfortably at this scale, offering:

- storage and search over billions of vectors,

- < 30 ms latency even under high-throughput Top-K retrieval,

- a fully distributed architecture that scales linearly as demand grows.

Whether you have a dozen agents or thousands working in parallel, Milvus keeps retrieval fast and consistent.


#### **3. Multi-Tenant Isolation**

Milvus provides granular multi-tenant isolation through **Partition Key**, a lightweight partitioning mechanism that segments memory inside a single collection. This allows:

- different teams, projects, or agent communities to maintain independent memory spaces,

- dramatically lower overhead compared to maintaining multiple collections,

- optional cross-partition retrieval when shared knowledge is needed.

This isolation is crucial for large multi-agent deployments where data boundaries must be respected without compromising retrieval speed.

OpenAgents connects to Milvus through **custom Mods** that call Milvus APIs directly. Agent messages, tool outputs, and interaction logs are automatically embedded into vectors and stored in Milvus. Developers can customize:

- the embedding model,

- storage schema and metadata,

- and retrieval strategies (e.g., hybrid search, partitioned search).

This gives each agent community a memory layer that is scalable, persistent, and optimized for semantic reasoning.

## How to Build a Multi-Agent Chatbot with OpenAgent and Milvus

To make things concrete, let’s walk through a demo: building a **developer-support community** where multiple specialist agents—Python experts, database experts, DevOps engineers, and more—collaborate to answer technical questions. Instead of relying on a single overworked generalist agent, each expert contributes domain-specific reasoning, and the system routes queries to the best-suited agent automatically.

This example demonstrates how to integrate **Milvus** into an OpenAgents deployment to provide long-term memory for technical Q&A. Agent conversations, past solutions, troubleshooting logs, and user queries are all converted into vector embeddings and stored in Milvus, giving the network the ability to:

- remember previous answers,

- reuse prior technical explanations,

- maintain consistency across sessions, and

- improve over time as more interactions accumulate.


### Prerequisite

- python3.11+

- conda

- Openai-key


### 1. Define Dependencies

Define the Python packages required for the project:

```
# Core framework
openagents>=0.6.11
# Vector database
pymilvus>=2.5.1
# Embedding model
sentence-transformers>=2.2.0
# LLM integration
openai>=1.0.0
# Environment config
python-dotenv>=1.0.0
```

### 2. Environment Variables

Here is the template for your environment configuration:

```
# LLM configuration (required)
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-4o
# Milvus configuration
MILVUS_URI=./multi_agent_memory.db
# Embedding model configuration
EMBEDDING_MODEL=text-embedding-3-large
EMBEDDING_DIMENSION=3072
# Network configuration
NETWORK_HOST=localhost
NETWORK_PORT=8700
STUDIO_PORT=8050
```

### 3. Configure Your OpenAgents Network

Define the structure of your agent network and its communication settings:

```
# Network transport protocol (HTTP on port 8700)
# Multi-channel messaging system (general, coordination, expert channels)
# Agent role definitions (coordinator, python_expert, etc.)
# Milvus integration settings
network:
  name: "Multi-Agent Collaboration Demo"
  transports:
    - type: "http"
      config:
        port: 8700
        host: "localhost"
  mods:
    - name: "openagents.mods.workspace.messaging"
      config:
        channels:
          - name: "general"          # User question channel
          - name: "coordination"     # Coordinator channel
          - name: "python_channel"   # Python expert channel
          - name: "milvus_channel"   # Milvus expert channel
          - name: "devops_channel"   # DevOps expert channel
  agents:
    coordinator:
      type: "coordinator"
      description: "Coordinator Agent, responsible for analyzing queries and dispatching tasks to expert agents"
      channels: ["general", "coordination"]
    python_expert:
      type: "expert"
      domain: "python"
```

### 4. Implement Multi-Agent Collaboration

The following shows core code snippets (not the full implementation).

```
# SharedMemory: Milvus’s SharedMemory system
# CoordinatorAgent: Coordinator Agent, responsible for analyzing queries and dispatching tasks to expert agents
# PythonExpertAgent: Python Expert
# MilvusExpertAgent: Milvus Expert
# DevOpsExpertAgent: DevOps Expert
import os
import asyncio
import json
from typing import List, Dict
from dotenv import load_dotenv
from openagents.agents.worker_agent import WorkerAgent
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
import openai
load_dotenv()
class SharedMemory:
    """SharedMemory in Milvus for all Agents"""
    def __init__(self):
        connections.connect(uri="./multi_agent_memory.db")
        self.setup_collections()
        self.openai_client = openai.OpenAI(
            api_key=os.getenv("OPENAI_API_KEY"),
            base_url=os.getenv("OPENAI_BASE_URL")
        )
    def setup_collections(self):
        """Create memory collections: expert knowledge, collaboration history, problem solutions"""
        collections = {
            "expert_knowledge": "expert knowledge",
            "collaboration_history": "collaboration history", 
            "problem_solutions": "problem solutions"
        }
        # Code to create vector collections...
    async def search_knowledge(self, query: str, collection_name: str):
        """Search for relevant knowledge"""
        # Vector search implementation...
    async def store_knowledge(self, agent_id: str, content: str, metadata: dict, collection_name: str):
        """Store knowledge"""
        # Store into the vector database...
class CoordinatorAgent(WorkerAgent):
    """Coordinator Agent - analyzes questions and coordinates other Agent"""
    def __init__(self):
        super().__init__(agent_id="coordinator")
        self.expert_agents = {
            "python": "python_expert",
            "milvus": "milvus_expert", 
            "devops": "devops_expert"
        }
    async def analyze_question(self, question: str) -> List[str]:
        """Determine which experts are needed for the question"""
        keywords = {
            "python": ["python", "django", "flask", "async"],
            "milvus": ["milvus", "vector", "index", "performance"],
            "devops": ["deployment", "docker", "kubernetes", "operations"]
        }
        # Keyword matching logic...
        return needed_experts
    async def coordinate_experts(self, question: str, needed_experts: List[str]):
        """Coordinate collaboration among expert Agent"""
        # 1. Notify experts to begin collaborating
        # 2. Dispatch tasks to each expert
        # 3. Collect expert responses
        # 4. Return expert opinions
    async def on_channel_post(self, context):
        """Main logic for handling user questions"""
        content = context.incoming_event.payload.get('content', {}).get('text', '')
        if content and not content.startswith('🎯'):
            # 1. Analyze question → 2. Coordinate experts → 3. Merge answers → 4. Reply to user
class PythonExpertAgent(WorkerAgent):
    """Python Expert Agent"""
    async def analyze_python_question(self, question: str) -> str:
        """Analyze Python-related questions and provide expert advice"""
        # 1. Search for relevant experience
        # 2. Use LLM to generate expert response
        # 3. Store result in collaboration history
        return answer
# Start all Agens
async def run_multi_agent_demo():
    coordinator = CoordinatorAgent()
    python_expert = PythonExpertAgent()
    milvus_expert = MilvusExpertAgent()
    devops_expert = DevOpsExpertAgent()
    # Connect to the OpenAgents network
    await coordinator.async_start(network_host="localhost", network_port=8700)
    # ... Start other Agent
    while True:
        await asyncio.sleep(1)
if __name__ == "__main__":
    asyncio.run(run_multi_agent_demo())
```

### 5. Create and Activate a Virtual Environment

```
conda create -n openagents
conda activate openagents
```

**Install Dependencies**

```
pip install -r requirements.txt
```

**Configure API Keys**

```
cp .env.example .env
```

**Start the OpenAgents Network**

```
openagents network start .
```

![](https://assets.zilliz.com/network_169812ab94.png)

**Start the Multi-Agent Service**

```
python multi_agent_demo.py
```

![](https://assets.zilliz.com/multiagent_service_1661d4b91b.png)

**Start OpenAgents Studio**

```
openagents studio -s
```

![](https://assets.zilliz.com/start_studio_4cd126fea2.png)

**Access Studio**

```
http://localhost:8050
```

![](https://assets.zilliz.com/access_studio1_a33709914b.png)

![](https://assets.zilliz.com/access_studio3_293604c79e.png)

![](https://assets.zilliz.com/access_studio_3_8d98a4cfe8.png)

**Check the status of your agents and network:**

![](https://assets.zilliz.com/check_state_bba1a4fe16.png)

## Conclusion

OpenAgents provides the coordination layer that lets agents discover each other, communicate, and collaborate, while Milvus solves the equally critical problem of how knowledge is stored, shared, and reused. By delivering a high-performance vector memory layer, Milvus enables agents to build persistent context, recall past interactions, and accumulate expertise over time. Together, they push AI systems beyond the limits of isolated models and toward the deeper collaborative potential of a true multi-agent network. 

Of course, no multi-agent architecture is without trade-offs. Running agents in parallel can increase token consumption, errors may cascade across agents, and simultaneous decision-making can lead to occasional conflicts. These are active areas of research and ongoing improvement—but they don’t diminish the value of building systems that can coordinate, remember, and evolve.

🚀 Ready to give your agents long-term memory?

Explore [Milvus](https://milvus.io/) and try integrating it with your own workflow. 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Build Smarter Multi-Agent Systems with OpenAgents and Milvus

OpenAgents x Milvus: How to Build Smarter Multi-Agent Systems That Share Memory


Gemini 3 Pro + Milvus: Robust RAG With Advanced Reasoning and Multimodal Power


Gemini 3 Pro + Milvus: Building a More Robust RAG With Advanced Reasoning and Multimodal Power



Over the past few weeks, a surprisingly heated argument has erupted across X and Hacker News: _Do we actually need MCP servers anymore?_ Some developers claim MCP is over-engineered, token-hungry, and fundamentally misaligned with how agents should use tools. Others defend MCP as the reliable way to expose real-world capabilities to language models. Depending on which thread you read, MCP is either the future of tool use—or dead on arrival.

![](https://assets.zilliz.com/hackernews_c3236cca2c.png)

The frustration is understandable. MCP gives you robust access to external systems, but it also forces the model to load long schemas, verbose descriptions, and sprawling tool lists. That adds real cost. If you download a meeting transcript and later feed it into another tool, the model may reprocess the same text multiple times, inflating token usage for no obvious benefit. For teams operating at scale, this isn’t an inconvenience—it’s a bill.

But declaring MCP obsolete is premature. Anthropic—the same team that invented MCP—quietly introduced something new: [**Skills**](https://claude.com/blog/skills). Skills are lightweight Markdown/YAML definitions that describe _how_ and _when_ a tool should be used. Instead of dumping full schemas into the context window, the model first reads compact metadata and uses that to plan. In practice, Skills dramatically reduce token overhead and give developers more control over tool orchestration.

So, does this mean Skills will replace MCP? Not quite. Skills streamline planning, but MCP still provides the actual capabilities: reading files, calling APIs, interacting with storage systems, or plugging into external infrastructure like [**Milvus**](https://milvus.io/), an open-source vector database that underpins fast semantic retrieval at scale, making it a critical backend when your Skills need real data access.

This post breaks down what Skills do well, where MCP still matters, and how both fit into Anthropic’s evolving agent architecture. Then we’ll walk through how to build your own Skills that integrate cleanly with Milvus.

## What Are Anthropic Agent Skills and How Do They Work

A long-standing pain point of traditional AI agents is that instructions get washed out as the conversation grows.

Even with the most carefully crafted system prompts, the model’s behavior can gradually drift over the course of the conversation. After several turns, Claude begins to forget or lose focus on the original instructions.

The problem lies in the structure of the system prompt. It is a one-time, static injection that competes for space in the model’s context window, alongside conversation history, documents, and any other inputs. As the context window fills, the model’s attention to the system prompt becomes increasingly diluted, leading to a loss of consistency over time. 

Skills were designed to address this issue. Skills are folders containing instructions, scripts, and resources. Rather than relying on a static system prompt, Skills break down expertise into modular, reusable, and persistent instruction bundles that Claude can discover and load dynamically when needed for a task.

When Claude begins a task, it first performs a lightweight scan of all available Skills by reading only their YAML metadata (just a few dozen tokens). This metadata provides just enough information for Claude to determine if a Skill is relevant to the current task. If so, Claude expands into the full set of instructions (usually under 5k tokens), and additional resources or scripts are loaded only if necessary. 

This progressive disclosure allows Claude to initialize a Skill with just 30–50 tokens, significantly improving efficiency and reducing unnecessary context overhead.

![](https://assets.zilliz.com/how_skills_works_a8563f346c.png)

## How Skills Compares to Prompts, Projects, MCP, and Subagents

Today’s model tooling landscape can feel crowded. Even within Claude's agentic ecosystem alone, there are several distinct components: Skills, prompts, Projects, subagents, and MCP.

Now that we understand what Skills are and how they work through modular instruction bundles and dynamic loading, we need to know how Skills relate to other parts of the Claude ecosystem, especially MCP. Here is a summary:

### 1. Skills

Skills are folders that contain instructions, scripts, and resources. Claude discovers and loads them dynamically using progressive disclosure: first metadata, then full instructions, and finally any required files.

**Best for:**

- Organizational workflows (brand guidelines, compliance procedures)

- Domain expertise (Excel formulas, data analysis)

- Personal preferences (note-taking systems, coding patterns)

- Professional tasks that need to be reused across conversations (OWASP-based code security reviews)

### 2. Prompts

Prompts are the natural-language instructions you give Claude within a conversation. They are temporary and exist only in the current conversation.

**Best for:**

- One-off requests (summarizing an article, formatting a list)

- Conversational refinement (adjusting tone, adding details)

- Immediate context (analyzing specific data, interpreting content)

- Ad-hoc instructions

### 3. Projects

Projects are self-contained workspaces with their own chat histories and knowledge bases. Each project offers a 200K context window. When your project knowledge approaches context limits, Claude transitions seamlessly into RAG mode, allowing up to a 10x expansion in effective capacity.

**Best for:**

- Persistent context (e.g., all conversations related to a product launch)

- Workspace organization (separate contexts for different initiatives)

- Team collaboration (on Team and Enterprise plans)

- Custom instructions (project-specific tone or perspective)

### 4. Subagents

Subagents are specialized AI assistants with their own context windows, custom system prompts, and specific tool permissions. They can work independently and return results to the main agent.

**Best for:**

- Task specialization (code review, test generation, security audits)

- Context management (keep the main conversation focused)

- Parallel processing (multiple subagents working on different aspects simultaneously)

- Tool restriction (e.g., read-only access)

### 5. MCP (Model Context Protocol)

The Model Context Protocol (MCP) is an open standard that connects AI models to external tools and data sources.

**Best for:**

- Accessing external data (Google Drive, Slack, GitHub, databases)

- Using business tools (CRM systems, project management platforms)

- Connecting to development environments (Local files, IDEs, version control)

- Integrating with custom systems (proprietary tools and data sources)

Based on the above, we can see that Skills and MCP address different challenges and work together to complement each other.

| **Dimension**           | **MCP**                                                        | **Skills**                                                              |
| ----------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------- |
| **Core Value**          | Connects to external systems (databases, APIs, SaaS platforms) | Defines behavior specifications (how to process and present data)       |
| **Questions Answered**  | "What can Claude access?"                                      | "What should Claude do?"                                                |
| **Implementation**      | Client-server protocol + JSON Schema                           | Markdown file + YAML metadata                                           |
| **Context Consumption** | Tens of thousands of tokens (multiple server accumulations)    | 30-50 tokens per operation                                              |
| **Use Cases**           | Querying large databases, calling GitHub APIs                  | Defining search strategies, applying filtering rules, output formatting |

Let's take code search as an example.

- **MCP (e.g., claude-context):** Provides the ability to access the Milvus vector database.

- **Skills:** Defines the workflow, such as prioritizing the most recently modified code, sorting results by relevance, and presenting the data in a Markdown table.

MCP provides the capability, while Skills define the process. Together, they form a complementary pair.

## How to Build Custom Skills with Claude-Context and Milvus

[Claude-Context](https://github.com/zilliztech/claude-context) is an MCP plugin that adds semantic code search functionality to Claude Code, turning the entire codebase into Claude’s context.

### Prerequisite

System Requirements:

- **Node.js**: Version >= 20.0.0 and < 24.0.0

- **OpenAI API Key** (for embedding models)

- [**Zilliz Cloud**](https://zilliz.com.cn/) **API Key** (managed Milvus service)

### Step 1: Configure the MCP Service (claude-context)

Run the following command in the terminal:

```
claude mcp add claude-context \
  -e OPENAI_API_KEY=sk-your-openai-api-key \
  -e MILVUS_ADDRESS=https://xxxxxxxxx-cn-hangzhou.cloud.zilliz.com.cn \
  -e MILVUS_TOKEN=your-zilliz-cloud-api-key \
  -e COLLECTION_NAME=medium_articles \
  -- npx @zilliz/claude-context-mcp@latest
```

Check the Configuration:

```
claude mcp list
```

![](https://assets.zilliz.com/Check_the_Configuration_5d2abd5ee8.webp)

The MCP setup is complete. Claude can now access the Milvus vector database.

### Step 2: Create the Skill

Create the Skills directory:

```
mkdir -p ~/.claude/skills/milvus-code-search
cd ~/.claude/skills/milvus-code-search
```

Create the SKILL.md file:

```
---
name: milvus-code-search
description: A semantic code search and architecture analysis skill designed for the Milvus codebase
---

## Instructions
When the user asks questions related to the Milvus codebase, I will:

1. **Code Search** : Use semantic search to locate relevant code snippets across the Milvus repository  
2. **Architecture Analysis** : Analyze Milvus’s module structure, component relationships, and design patterns  
3. **Feature Explanation** : Explain how specific features are implemented and how the corresponding logic works  
4. **Development Guidance** : Provide suggestions, best practices, and improvement ideas for modifying the code  

## Target Repository
- **Core Modules**:  
  - `internal/` — Core internal components  
  - `pkg/` — Public packages and utilities  
  - `client/` — Go client implementation  
  - `cmd/` — Command-line tools  

## Usage Examples

### Architecture Query
User: How does Milvus’s query coordinator work?  
Assistant: *(searching for `querycoordv2`)* Let me walk you through how the query coordinator operates in Milvus…

### Feature Implementation
User: How does Milvus implement vector indexing?  
Assistant: *(searching for `index` code)* The vector indexing logic in Milvus is mainly implemented in the following modules…

### Code Understanding
User: What does this function do? *(points to a specific file)*  
Assistant: *(analyzing the surrounding code)* Based on the context of the Milvus codebase, this function is responsible for…

### Development Guidance
User: How can I add a new vector distance metric to Milvus?  
Assistant: *(searching for `distance` implementations)* Following the existing pattern, you can add a new distance method by…

## Best Practices
1. **Precise Search** : Use specific technical terms and module names  
2. **Contextual Understanding** : Interpret code within Milvus’s overall system architecture  
3. **Actionable Advice** : Provide practical, implementation-ready suggestions  
4. **Performance Awareness** : Consider Milvus’s requirements as a high-performance vector database  

---

*A custom code-search Skill tailored for the open-source Milvus vector database project.*

---
```

### Step 3: Restart Claude to Apply Skills

Run the following command to restart Claude:

```
claude
```

**Note:** After the configuration is complete, you can immediately use the Skills to query the Milvus codebase.

Below is an example of how it works. 

Query: How does Milvus QueryCoord work?

![](https://assets.zilliz.com/code_a95429ddb0.png)

![](https://assets.zilliz.com/code2_d58a942777.png)

![](https://assets.zilliz.com/code3_6c9f835c65.png)

## Conclusion

At its core, Skills act as a mechanism for encapsulating and transferring specialized knowledge. By using Skills, AI can inherit a team's experience and follow industry best practices—whether that’s a checklist for code reviews or documentation standards. When this tacit knowledge is made explicit through Markdown files, the quality of AI-generated outputs can see a significant improvement.

Looking ahead, the ability to leverage Skills effectively could become a key differentiator in how teams and individuals use AI to their advantage. 

As you explore the potential of AI in your organization, Milvus stands as a critical tool for managing and searching large-scale vector data. By pairing Milvus’ powerful vector database with AI tools like Skills, you can improve not only your workflows but also the depth and speed of your data-driven insights.

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) to chat with our engineers and other AI engineers in the community. You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).

Exploring Skills, MCP, and Milvus for Smarter AI Workflows

Is MCP Already Outdated? The Real Reason Anthropic Shipped Skills—and How to Pair Them with Milvus



*This post was originally published on [Cloudian](https://cloudian.com/blog/cloudian-deploys-new-nvidia-rdma-for-s3-compatible-storage-a-game-changer-for-ai-storage-performance/) and is reposted here with permission.*

Cloudian has collaborated with NVIDIA to add support for RDMA for S3-compatible storage to its HyperStore® solution, drawing on its 13+ years of experience in S3 API implementation. As an S3-API based platform with parallel processing architecture, Cloudian is uniquely suited to both contribute to and capitalize on the development of this technology. This collaboration leverages Cloudian’s deep expertise in object storage protocols and NVIDIA’s leadership in compute and network acceleration to create a solution that seamlessly integrates high-performance computing with enterprise-scale storage.

NVIDIA has announced the upcoming general availability of RDMA for S3-compatible storage (Remote Direct Memory Access) technology, marking a significant milestone in AI infrastructure evolution. This breakthrough technology promises to transform how organizations handle the massive data requirements of modern AI workloads, delivering unprecedented performance improvements while maintaining the scalability and simplicity that has made S3-compatible object storage the foundation of cloud computing.

## What is RDMA for S3-compatible storage?

This launch represents a fundamental advancement in how storage systems communicate with AI accelerators. The technology enables direct data transfers between S3 API-compatible object storage and GPU memory, completely bypassing traditional CPU-mediated data paths. Unlike conventional storage architectures that route all data transfers through the CPU and system memory—creating bottlenecks and latency—RDMA for S3-compatible storage establishes a direct highway from storage to GPU.

![](https://assets.zilliz.com/RDMA_for_S3_compatible_storage_18fdd69d02.webp)

At its core, this technology eliminates intermediate steps with a direct pathway that reduces latency, dramatically cuts CPU processing demands, and significantly reduces power consumption. The result is storage systems that can deliver data at the speed modern GPUs require for demanding AI applications.

The technology maintains compatibility with the ubiquitous S3 APIs while adding this high-performance data path. Commands are still issued through standard S3-API based storage protocols, but the actual data transfer occurs via RDMA directly to GPU memory, bypassing the CPU entirely and eliminating the overhead of TCP protocol processing.

## Breakthrough Performance Results

The performance improvements delivered by RDMA for S3-compatible storage are nothing short of transformational. Real-world testing demonstrates the technology’s ability to eliminate storage I/O bottlenecks that constrain AI workloads.

### Dramatic Speed Improvements:

- **35 GB/s per node throughput** (reads) measured, with linear scalability across clusters

- **Scalability to TBs/s** with Cloudian’s parallel processing architecture

- **3-5x throughput improvement** compared to conventional TCP-based object storage

### Resource Efficiency Gains:

- **90% reduction in CPU utilization** by establishing direct data pathways to GPUs

- **Increase GPU utilization** with bottlenecks eliminated

- Dramatic reduction in power consumption through reduced processing overhead

- Cost reductions for AI storage

### 8X Performance Boost on Milvus by Zilliz Vector DB

These performance improvements are particularly evident in vector database operations, where collaboration between Cloudian and Zilliz using [NVIDIA cuVS](https://developer.nvidia.com/cuvs) and [NVIDIA L40S GPUs](https://www.nvidia.com/en-us/data-center/l40s/) demonstrated an **8x performance boost in Milvus operations** when compared with CPU-based systems and TCP-based data transfer. This represents a fundamental shift from storage being a constraint to storage enabling AI applications to achieve their full potential.

![](https://assets.zilliz.com/vector_indexing_time_354bdd4b46.webp)

## Why S3 API-based Object Storage for AI Workloads

The convergence of RDMA technology  with object storage architecture creates the ideal foundation for AI infrastructure, addressing multiple challenges that have constrained traditional storage approaches.

**Exabyte Scalability for AI’s Data Explosion:** AI workloads, particularly those involving synthetic and multi-modal data, are pushing storage requirements into the 100-petabyte range and beyond. Object storage’s flat address space scales seamlessly from petabytes to exabytes, accommodating the exponential growth in AI training datasets without the hierarchical limitations that constrain file-based systems.

**Unified Platform for Complete AI Workflows:** Modern AI operations span data ingestion, cleansing, training, checkpointing, and inference—each with distinct performance and capacity requirements. S3-compatible object storage supports this entire spectrum through consistent API access, eliminating the complexity and cost of managing multiple storage tiers. Training data, models, checkpoint files, and inference datasets can all reside in a single, high-performance data lake.

**Rich Metadata for AI Operations:** Critical AI operations like search and enumeration are fundamentally metadata-driven. Object storage’s rich, customizable metadata capabilities enable efficient data tagging, searching, and management—essential for organizing and retrieving data in complex AI model training and inference workflows.

**Economic and Operational Advantages:** S3-compatible object storage delivers up to 80% lower total cost of ownership compared to file storage alternatives, leveraging industry-standard hardware and independent scaling of capacity and performance. This economic efficiency becomes crucial as AI datasets reach enterprise scale.

**Enterprise Security and Governance:** Unlike GPUDirect implementations that require kernel-level modifications, RDMA for S3-compatible storage requires no vendor-specific kernel changes, maintaining system security and regulatory compliance. This approach is particularly valuable in sectors like healthcare and finance where data security and regulatory compliance are paramount.

## The Road Ahead

NVIDIA’s announcement of RDMA for S3-compatible storage general availability represents more than a technological milestone—it signals the maturation of AI infrastructure architecture. By combining the limitless scalability of object storage with the breakthrough performance of direct GPU access, organizations can finally build AI infrastructures that scale with their ambitions.

As AI workloads continue to grow in complexity and scale, RDMA for S3-compatible storage provides the storage foundation that enables organizations to maximize their AI investments while maintaining data sovereignty and operational simplicity. The technology transforms storage from a bottleneck into an enabler, allowing AI applications to achieve their full potential at enterprise scale.

For organizations planning their AI infrastructure roadmap, the general availability of RDMA for S3-compatible storage marks the beginning of a new era where storage performance truly matches the demands of modern AI workloads.

## Industry Perspectives

As AI becomes increasingly central to healthcare delivery, we continuously seek to boost the performance and efficiency of our infrastructure. The new RDMA for S3-compatible storage from NVIDIA and Cloudian will be critical for our medical imaging analysis and diagnostic AI applications, where processing large datasets quickly can directly impact patient care, while reducing costs of moving data between S3-API based storage devices and SSD based NAS storages.  – _Dr. Swapnil Rane MD, DNB, PDCC (Nephropath), Mres (TCM), Fellowship in Oncopath, FRCPath Professor (F) of Pathology, PI, AI/Computational Pathology And Imaging Lab OIC- Department of Digital and Computational Oncology, Tata Memorial Centre_

“NVIDIA’s RDMA for S3-compatible announcement confirms the value of our Cloudian-based AI infrastructure strategy. We enable organizations to run high-performance AI at scale while preserving S3 API compatibility that keeps migration simple and application development costs low.” – _Sunil Gupta, Co-founder, Managing Director & Chief Executive Officer (CEO), Yotta Data Services_

“As we expand our on-premises capabilities to deliver sovereign AI, NVIDIA’s RDMA for S3-compatible storage technology and Cloudian’s high-performance object storage give us the performance we need without compromising data residency and without requiring any kernel-level modifications. The Cloudian HyperStore platform lets us scale to exabytes while keeping our sensitive AI data completely under our control.” – _Logan Lee, EVP & Head of Cloud, Kakao_

“We’re excited about NVIDIA’s announcement of the upcoming RDMA for S3-compatible storage GA release. Our testing with Cloudian showed up to 8X performance improvement for vector database operations, which will let our Milvus by Zilliz users achieve cloud-scale performance for demanding AI workloads while maintaining complete data sovereignty.” – _Charles Xie, Founder and CEO of Zilliz_


8× Milvus Boost with Cloudian HyperStore and NVIDIA RDMA

Unlocking 8× Milvus Performance with Cloudian HyperStore and NVIDIA RDMA for S3 Storage



*This post was originally published on [HPE Community](https://community.hpe.com/t5/around-the-storage-block/power-high-performance-rag-for-genai-with-hpe-alletra-storage-mp/ba-p/7257369) and is reposted here with permission.*

HPE Alletra Storage MP X10000 and Milvus power scalable, low-latency RAG, enabling LLMs to deliver accurate, context-rich responses with high-performance vector search for GenAI workloads.

## In generative AI, RAG needs more than just an LLM

Context unleashes the true power of generative AI (GenAI) and large language models (LLMs). When an LLM has the right signals to orient its responses, it can deliver answers that are accurate, relevant, and trustworthy.

Think of it this way: if you were dropped into a dense jungle with a GPS device but no satellite signal. The screen shows a map, but without your current position, it is useless for navigation. On the contrary, a GPS with a strong satellite signal doesn’t just show a map; it gives you turn-by-turn guidance.

That’s what retrieval-augmented generation (RAG) does for LLMs. The model already has the map (its pretrained knowledge), but not the direction (your domain-specific data). LLMs without RAG are like GPS devices that are full of knowledge but have no real-time orientation. RAG provides the signal that tells the model where it is and where to go.

RAG grounds model responses in trusted, up-to-date knowledge pulled from your own domain-specific content from policies, product docs, tickets, PDFs, code, audio transcripts, images, and more. Making RAG work at scale is challenging. The retrieval process needs to be fast enough to keep user experiences seamless, accurate enough to return the most relevant information, and predictable even when the system is under heavy load. That means handling high query volumes, ongoing data ingestion, and background tasks like index building without performance degradation. Spinning up a RAG pipeline with a few PDFs is relatively straightforward. However, when scaling to hundreds of PDFs, it becomes significantly more challenging. You can’t keep everything in memory, so a robust and efficient storage strategy becomes essential to manage embeddings, indexes, and retrieval performance. RAG requires a vector database and a storage layer that can keep pace as concurrency and data volumes grow.

## Vector databases power RAG

At the heart of RAG is semantic search, finding information by meaning rather than exact keywords. This is where vector databases come in. They store high-dimensional embeddings of text, images, and other unstructured data, enabling similarity search that retrieves the most relevant context for your queries. Milvus is a leading example: a cloud-native, open-source vector database built for billion-scale similarity search. It supports hybrid search, combining vector similarity with keyword and scalar filters for precision, and offers independent scaling of compute and storage with GPU-aware optimization options for acceleration. Milvus also manages data through a smart segment lifecycle, moving from growing to sealed segments with compaction and multiple approximate nearest neighbor (ANN) indexing options such as HNSW and DiskANN, ensuring performance and scalability for real-time AI workloads such as RAG.

## The hidden challenge: Storage throughput & latency

Vector search workloads put pressure on every part of the system. They demand high‑concurrency ingestion while maintaining low‑latency retrieval for interactive queries. At the same time, background operations such as index building, compaction, and data reloads must run without disrupting live performance. Many performance bottlenecks in traditional architectures trace back to storage. Whether it’s input/output (I/O) limitations, metadata lookup delays, or concurrency constraints. To deliver predictable, real‑time performance at scale, the storage layer must keep pace with the demands of vector databases.

## The storage foundation for high performance vector search

[HPE Alletra Storage MP X10000](https://www.hpe.com/in/en/alletra-storage-mp-x10000.html) is a flash-optimized, all-NVMe, S3-compatible object storage platform engineered for real-time performance at scale. Unlike traditional capacity-focused object stores, HPE Alletra Storage MP X10000 is designed for low-latency, high-throughput workloads like vector search. Its log-structured key-value engine and extent-based metadata enable highly parallel reads and writes, while GPUDirect RDMA provides zero-copy data paths that reduce CPU overhead and accelerate data movement to GPUs. The architecture supports disaggregated scaling, allowing capacity and performance to grow independently, and includes enterprise-grade features such as encryption, role-based access control (RBAC), immutability, and data durability. Combined with its cloud-native design, HPE Alletra Storage MP X10000 integrates seamlessly with Kubernetes environments, making it an ideal storage foundation for Milvus deployments.

## HPE Alletra Storage MP X10000 and Milvus: A scalable foundation for RAG

HPE Alletra Storage MP X10000 and Milvus complement each other to deliver RAG that is fast, predictable, and easy to scale. Figure 1 illustrates the architecture of scalable AI use cases and RAG pipelines, showing how Milvus components deployed in a containerized environment interact with high performance object storage from HPE Alletra Storage MP X10000.

![](https://assets.zilliz.com/Figure_1_Architecture_of_scalable_AI_use_cases_and_RAG_pipeline_using_HPE_Alletra_Storage_MP_X10000_and_Milvus_ed3a87a5ee.png)

Milvus cleanly separates compute from storage, while HPE Alletra Storage MP X10000 provides high‑throughput, low‑latency object access that keeps pace with vector workloads. Together, they enable predictable scale‑out performance: Milvus distributes queries across shards, and HPE Alletra Storage MP X10000’s fractional, multidimensional scaling keeps latency consistent as data and QPS grow. In simple terms, you add exactly the capacity or performance you need, when you need it. Operational simplicity is another advantage: HPE Alletra Storage MP X10000 sustains maximum performance from a single bucket, eliminating complex tiering, while enterprise features (encryption, RBAC, immutability, robust durability) support on‑prem or hybrid deployments with strong data sovereignty and consistent service-level objectives (SLOs).

When vector search scales, storage is often blamed for slow ingestion, compaction, or retrieval. With Milvus on HPE Alletra Storage MP X10000, that narrative changes. The platform’s all‑NVMe, log‑structured architecture and GPUDirect RDMA option deliver consistent, ultra‑low‑latency object access—even under heavy concurrency and during lifecycle operations like index build and reload. In practice, your RAG pipelines remain compute‑bound, not storage‑bound. As collections grow and query volumes spike, Milvus stays responsive while HPE Alletra Storage MP X10000 preserves I/O headroom, enabling predictable, linear scalability without re‑architecting storage. This becomes especially important as RAG deployments scale beyond initial proof-of-concept stages and move into full production.

## Enterprise-ready RAG: Scalable, predictable, and built for GenAI

For RAG and real-time GenAI workloads, the combination of HPE Alletra Storage MP X10000 and Milvus delivers a future-ready foundation that scales with confidence. This integrated solution empowers organizations to build intelligent systems that are fast, elastic, and secure—without compromising on performance or manageability. Milvus provides distributed, GPU-accelerated vector search with modular scaling, while HPE Alletra Storage MP X10000 ensures ultrafast, low-latency object access with enterprise-grade durability and lifecycle management. Together, they decouple compute from storage, enabling predictable performance even as data volumes and query complexity grow. Whether you’re serving real-time recommendations, powering semantic search, or scaling across billions of vectors, this architecture keeps your RAG pipelines responsive, cost-efficient, and cloud-optimized. With seamless integration into Kubernetes and HPE GreenLake cloud, you gain unified management, consumption-based pricing, and the flexibility to deploy across hybrid or private cloud environments. HPE Alletra Storage MP X10000 and Milvus: a scalable, high performance RAG solution built for the demands of modern GenAI.


Optimized RAG with HPE Alletra Storage MP X10000 + Milvus

Power high performance RAG for GenAI with HPE Alletra Storage MP + Milvus


How Parlant × Milvus Brings Control to LLM Agent Behavior

Beyond Context Overload: How Parlant × Milvus Brings Control and Clarity to LLM Agent Behavior



Building an AI workflow often feels harder than it should. Between writing glue code, debugging API calls, and managing data pipelines, the process can eat up hours before you even see results. [**Langflow**](https://www.langflow.org/) and [**Milvus**](https://milvus.io/) simplify this dramatically — giving you a code-light way to design, test, and deploy retrieval-augmented generation (RAG) workflows in minutes, not days.

**Langflow** offers a clean, drag-and-drop interface that feels more like sketching ideas on a whiteboard than coding. You can visually connect language models, data sources, and external tools to define your workflow logic — all without touching a line of boilerplate code.

Paired with **Milvus**, the open-source vector database that gives LLMs long-term memory and contextual understanding, the two form a complete environment for production-grade RAG. Milvus efficiently stores and retrieves embeddings from your enterprise or domain-specific data, allowing LLMs to generate answers that are grounded, accurate, and context-aware.

In this guide, we’ll walk through how to combine Langflow and Milvus to build an advanced RAG workflow — all through a few drags, drops, and clicks.


## What is Langflow? 

Before going through the RAG demo, let’s learn what Langflow is and what it can do. 

Langflow is an open-source, Python-based framework that makes it easier to build and experiment with AI applications. It supports key AI capabilities such as agents and the Model Context Protocol (MCP), giving developers and non-developers alike a flexible foundation for creating intelligent systems. 

At its core, Langflow provides a visual editor. You can drag, drop, and connect different resources to design complete applications that combine models, tools, and data sources. When you export a workflow, Langflow automatically generates a file named `FLOW_NAME.json` on your local machine. This file records all the nodes, edges, and metadata that describe your flow, allowing you to version-control, share, and reproduce projects easily across teams.

![](https://assets.zilliz.com/Langflow_s_visual_editor_cd553ad4ad.png)

Behind the scenes, a Python-based runtime engine executes the flow. It orchestrates LLMs, tools, retrieval modules, and routing logic — managing data flow, state, and error handling to ensure smooth execution from start to finish.

Langflow also includes a rich component library with prebuilt adapters for popular LLMs and vector databases — including [Milvus](https://milvus.io/). You can extend this further by creating custom Python components for specialized use cases. For testing and optimization, Langflow offers step-by-step execution, a Playground for rapid testing, and integrations with LangSmith and Langfuse for monitoring, debugging, and replaying workflows end-to-end.


## Hands-on Demo: How to Build a RAG Workflow with Langflow and Milvus

Building on Langflow’s architecture, Milvus can serve as the vector database that manages embeddings and retrieves private enterprise data or domain-specific knowledge.

In this demo, we’ll use Langflow’s Vector Store RAG template to demonstrate how to integrate Milvus and build a vector index from local data, enabling efficient, context-enhanced question answering.

![](https://assets.zilliz.com/data_processing_flow_289a9376c9.webp)

### Prerequisites：

1.Python 3.11 (or Conda)

2.uv

3.Docker & Docker Compose

4.OpenAI key


### Step 1. Deploy Milvus Vector Database

Download the deployment files.
```
wget <https://github.com/Milvus-io/Milvus/releases/download/v2.5.12/Milvus-standalone-docker-compose.yml> -O docker-compose.yml
```
Start the Milvus service.
```
docker-compose up -d
```
```
docker-compose ps -a
```

![](https://assets.zilliz.com/start_milvus_service_860353ed55.webp)

### Step 2. Create a Python Virtual Environment
```
conda create -n langflow
# activate langflow and launch it
conda activate langflow
```
### Step 3. Install the Latest Packages
```
pip install langflow -U
```
### Step 4. Launch Langflow
```
uv run langflow run
```
Visit Langflow.
```
<http://127.0.0.1:7860>
```
### Step 5. Configure the RAG Template
Select the Vector Store RAG template in Langflow.

![](https://assets.zilliz.com/rag1_fcb0d1c3c5.webp)

![](https://assets.zilliz.com/rag2_f750e10a41.webp)

Choose Milvus as your default vector database.

![](https://assets.zilliz.com/vdb_milvus_925c6ce846.webp)

In the left panel, search for “Milvus” and add it to your flow.

![](https://assets.zilliz.com/add_milvus1_862d14d0d0.webp)

![](https://assets.zilliz.com/add_milvus2_4e3d6aacda.webp)

Configure Milvus connection details. Leave other options as the default for now.

![](https://assets.zilliz.com/connect1_a27d3e4f43.webp)

![](https://assets.zilliz.com/connect2_d8421c1525.webp)

Add your OpenAI API key to the relevant node.

![](https://assets.zilliz.com/openai_key_7a6596868c.webp)

![](https://assets.zilliz.com/openai_key2_4753bfb4d0.webp)

### Step 6. Prepare Test Data
Note: Use the official FAQ for Milvus 2.6 as the test data.
```
https://github.com/milvus-io/milvus-docs/blob/v2.6.x/site/en/faq/product_faq.md
```
### Step 7. Phase One Testing
Upload your dataset and ingest it into Milvus.
Note: Langflow then converts your text into vector representations. You must upload at least two datasets, or the embedding process will fail. This is a known bug in Langflow’s current node implementation.

![](https://assets.zilliz.com/ingest_7b804d870a.webp)

![](https://assets.zilliz.com/ingest2_fc7f1e4d9a.webp)

Check the status of your nodes.

![](https://assets.zilliz.com/test_48e02d48ca.webp)

### Step 8. Phase Two Testing

![](https://assets.zilliz.com/ingest_7b804d870a.webp)

### Step 9. Run the Full RAG Workflow

![](https://assets.zilliz.com/full_flow1_5b4f4962f5.webp)

![](https://assets.zilliz.com/full_flow2_535c722a3d.webp)

## Conclusion

Building AI workflows doesn’t have to be complicated. Langflow + Milvus makes it fast, visual, and code-light — a simple way to enhance RAG without heavy engineering effort.

Langflow’s drag-and-drop interface makes it a suitable choice for teaching, workshops, or live demos, where you need to demonstrate how AI systems work in a clear and interactive manner. For teams seeking to integrate intuitive workflow design with enterprise-grade vector retrieval, combining Langflow’s simplicity with Milvus’s high-performance search delivers both flexibility and power.

👉 Start building smarter RAG workflows with [Milvus](https://milvus.io/) today.

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute one-on-one session to get insights, guidance, and answers to your questions through[ Milvus Office Hours](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


Drag, Drop, and Deploy RAG Workflows with Langflow & Milvus

Drag, Drop, and Deploy: How to Build RAG Workflows with Langflow and Milvus



In a vector database, we often need to quickly find the most similar results among vast collections of high-dimensional vectors—such as image features, text embeddings, or audio representations. Without an index, the only option is to compare the query vector against every single vector in the dataset. This **brute-force search** might work when you have a few thousand vectors, but once you’re dealing with tens or hundreds of millions, it becomes unbearably slow and computationally expensive. 

That’s where **Approximate Nearest Neighbor (ANN)** search comes in. Think of it like looking for a specific book in a massive library. Instead of checking every book one by one, you start by browsing the sections most likely to contain it. You might not get the _exact_ same results as a full search, but you’ll get very close—and in a fraction of the time. In short, ANN trades a slight loss in accuracy for a significant boost in speed and scalability.

Among the many ways to implement ANN search, **IVF (Inverted File)** and **HNSW (Hierarchical Navigable Small World)** are two of the most widely used. But IVF stands out for its efficiency and adaptability in large-scale vector search. In this article, we’ll walk you through how IVF works and how it compares with HNSW—so you can understand their trade-offs and choose the one that best fits your workload.

## What is an IVF Vector Index?

**IVF (Inverted File)** is one of the most widely used algorithms for ANN. It borrows its core idea from the “inverted index” used in text retrieval systems—only this time, instead of words and documents, we’re dealing with vectors in a high-dimensional space.

Think of it like organizing a massive library. If you dumped every book (vector) into one giant pile, finding what you need would take forever. IVF solves this by first **clustering** all vectors into groups, or _buckets_. Each bucket represents a “category” of similar vectors, defined by a **centroid**—a kind of summary or “label” for everything inside that cluster.

When a query comes in, the search happens in two steps:

**1. Find the nearest clusters.** The system looks for the few buckets whose centroids are closest to the query vector—just like heading straight to the two or three library sections most likely to have your book.

**2. Search within those clusters.** Once you’re in the right sections, you only need to look through a small set of books instead of the entire library.

This approach cuts down the amount of computation by orders of magnitude. You still get highly accurate results—but much faster. 


## How to Build an IVF Vector Index

The process of building an IVF vector index involves three main steps: K-means Clustering, Vector Assignment, and Compression Encoding (Optional). The full process looks like this:

![](https://assets.zilliz.com/ivf_building_process_90c2966975.webp)

### Step 1: K-means Clustering

First, run k-means clustering on the dataset X to divide the high-dimensional vector space into nlist clusters. Each cluster is represented by a centroid, which is stored in the centroid table C. The number of centroids, nlist, is a key hyperparameter that determines how fine-grained the clustering will be.

Here’s how k-means works under the hood:

- **Initialization:** Randomly select _nlist_ vectors as the initial centroids.

- **Assignment:** For each vector, compute its distance to all centroids and assign it to the nearest one.

- **Update:** For each cluster, calculate the average of its vectors and set that as the new centroid.

- **Iteration and convergence:** Repeat assignment and update until centroids stop changing significantly or a maximum number of iterations is reached.

Once k-means converges, the resulting nlist centroids form the “index directory” of IVF. They define how the dataset is coarsely partitioned, allowing queries to quickly narrow down the search space later on.

Think back to the library analogy: training centroids is like deciding how to group books by topic:

- A larger nlist means more sections, each with fewer, more specific books.

- A smaller nlist means fewer sections, each covering a broader, more mixed range of topics.

### Step 2: Vector Assignment

Next, each vector is assigned to the cluster whose centroid it’s closest to, forming inverted lists (List_i). Each inverted list stores the IDs and storage information of all vectors that belong to that cluster. 

You can think of this step like shelving the books into their respective sections. When you’re looking for a title later, you only need to check the few sections that are most likely to have it, instead of wandering the whole library.

### Step 3: Compression Encoding (Optional)

To save memory and speed up computation, vectors within each cluster can go through compression encoding. There are two common approaches:

- **SQ8 (Scalar Quantization):** This method quantizes each dimension of a vector into 8 bits. For a standard `float32` vector, each dimension typically takes up 4 bytes. With SQ8, it’s reduced to just 1 byte—achieving a 4:1 compression ratio while keeping the vector’s geometry largely intact.

- **PQ (Product Quantization):** It splits a high-dimensional vector into several subspaces. For example, a 128-dimensional vector can be divided into 8 sub-vectors of 16 dimensions each. In each subspace, a small codebook (typically with 256 entries) is pre-trained, and each sub-vector is represented by an 8-bit index pointing to its nearest codebook entry. This means the original 128-D `float32` vector (which requires 512 bytes) can be represented using only 8 bytes (8 subspaces × 1 byte each), achieving a 64:1 compression ratio.

## How to Use the IVF Vector Index for Search

Once the centroid table, inverted lists, and the compression encoder and codebooks (optional) are built, the IVF index can be used to accelerate similarity search. The process typically has three main steps, as shown below:

![](https://assets.zilliz.com/ivf_search_process_025d3f444f.webp)

### Step 1: Calculate distances from the query vector to all centroids

When a query vector q arrives, the system first determines which clusters it is most likely to belong to. Then, it computes the distance between q and every centroid in the centroid table C—usually using Euclidean distance or inner product as the similarity metric. The centroids are then sorted by their distance to the query vector, producing an ordered list from nearest to farthest.

For example, as shown in the illustration, the order is: C4 < C2 < C1 < C3 < C5.


### Step 2: Select the nearest nprobe clusters

To avoid scanning the entire dataset, IVF only searches within the top _nprobe_ clusters that are closest to the query vector.

The parameter nprobe defines the search scope and directly affects the balance between speed and recall:

- A smaller nprobe leads to faster queries but may reduce recall.

- A larger nprobe improves recall but increases latency.

In real-world systems, nprobe can be dynamically tuned based on the latency budget or accuracy requirements.
In the example above, if nprobe = 2, the system will only search within Cluster 2 and Cluster 4—the two nearest clusters.


### Step 3: Search the nearest neighbor in the selected clusters

Once the candidate clusters are selected, the system compares the query vector q with the vectors stored inside them.
&#x20;There are two main modes of comparison:

- **Exact Comparison (IVF_FLAT)**: The system retrieves the original vectors from the selected clusters and computes their distances to q directly, returning the most accurate results.

- **Approximate Comparison (IVF_PQ / IVF_SQ8)**: When PQ or SQ8 compression is used, the system employs a **lookup table method** to accelerate distance computation. Before the search begins, it precomputes the distances between the query vector and each codebook entry. Then, for each vector, it can simply “look up and sum” these precomputed distances to estimate similarity.

Finally, the candidate results from all searched clusters are merged and re-ranked, producing the Top-k most similar vectors as the final output.


## IVF In-Practice

Once you understand how IVF vector indexes are **built** and **searched**, the next step is to apply them for real-world workloads. In practice, you’ll often need to balance **performance**, **accuracy**, and **memory usage**. Below are some practical guidelines drawn from engineering experience.


### How to Choose the Right nlist

As mentioned earlier, the parameter nlist determines the number of clusters into which the dataset is divided when building an IVF index. 

- **Larger nlist**: Creates finer-grained clusters, meaning each cluster contains fewer vectors. This reduces the number of vectors scanned during search and generally results in faster queries. But building the index takes longer, and the centroid table consumes more memory.

- **Smaller nlist**: Speeds up index construction and reduces memory usage, but each cluster becomes more “crowded.” Each query must scan more vectors within a cluster, which can lead to performance bottlenecks.

Based on these trade-offs, here’s a practical rule of thumb: 

For datasets at the **million scale**, a good starting point is **nlist ≈ √n** (n is the number of vectors in the data shard being indexed).

For example, if you have 1 million vectors, try nlist = 1,000. For larger datasets—tens or hundreds of millions—most vector databases shard the data so that each shard contains around one million vectors, keeping this rule practical.

Because nlist is fixed at index creation, changing it later means rebuilding the entire index. So it’s best to experiment early. Test several values—ideally in powers of two (e.g., 1024, 2048)—to find the sweet spot that balances speed, accuracy, and memory for your workload.


### How to Tune nprobe

The parameter nprobe controls the number of clusters searched during query time. It directly affects the trade-off between recall and latency.

- **Larger nprobe**: Covers more clusters, leading to higher recall but also higher latency. The delay generally increases linearly with the number of clusters searched.

- **Smaller nprobe**: Searches fewer clusters, resulting in lower latency and faster queries. However, it may miss some true nearest neighbors, slightly lowering recall and result accuracy.

If your application is not extremely sensitive to latency, it’s a good idea to experiment with nprobe dynamically—for example, testing values from 1 to 16 to observe how recall and latency change. The goal is to find the sweet spot where recall is acceptable and latency remains within your target range.

Since nprobe is a runtime search parameter, it can be adjusted on the fly without requiring the index to be rebuilt. This enables fast, low-cost, and highly flexible tuning across different workloads or query scenarios.

### Common Variants of the IVF Index

When building an IVF index, you’ll need to decide whether to use compression encoding for the vectors in each cluster—and if so, which method to use.

This results in three common IVF index variants: 

| **IVF Variant** | **Key Features**                                                                                                                                      | **Use Cases**                                                                                                                                                                                                                               |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **IVF_FLAT**   | Stores raw vectors within each cluster without compression. Offers the highest accuracy, but also consumes the most memory.                           | Ideal for medium-scale datasets (up to hundreds of millions of vectors) where high recall (95%+) is required.                                                                                                                               |
| **IVF_PQ**     | Applies Product Quantization (PQ) to compress vectors within clusters. By adjusting the compression ratio, memory usage can be significantly reduced. | Suitable for large-scale vector search (hundreds of millions or more) where some accuracy loss is acceptable. With a 64:1 compression ratio, recall is typically around 70%, but can reach 90% or higher by lowering the compression ratio. |
| **IVF_SQ8**    | Uses Scalar Quantization (SQ8) to quantize vectors. Memory usage sits between IVF_FLAT and IVF_PQ.                                                  | Ideal for large-scale vector search where you need to maintain relatively high recall (90%+) while improving efficiency.                                                                                                                    |


## IVF vs HNSW: Pick What Fits

Besides IVF, **HNSW (Hierarchical Navigable Small World)** is another widely used in-memory vector index. The table below highlights the key differences between the two.

|                                  | **IVF**                                                                                                                        | **HNSW**                                                                                                                                                                      |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Algorithm Concept**            | Clustering and bucketing                                                                                                       | Multi-layer graph navigation                                                                                                                                                  |
| **Memory Usage**                 | Relatively low                                                                                                                 | Relatively high                                                                                                                                                               |
| **Index Build Speed**            | Fast (only requires clustering)                                                                                                | Slow (needs multi-layer graph construction)                                                                                                                                   |
| **Query Speed (No Filtering)**   | Fast, depends on _nprobe_                                                                                                      | Extremely fast, but with logarithmic complexity                                                                                                                               |
| **Query Speed (With Filtering)** | Stable — performs coarse filtering at the centroid level to narrow down candidates                                             | Unstable — especially when the filtering ratio is high (90%+), the graph becomes fragmented and may degrade to near full-graph traversal, even slower than brute-force search |
| **Recall Rate**                  | Depends on whether compression is used; without quantization, recall can reach **95%+**                                        | Usually higher, around **98%+**                                                                                                                                               |
| **Key Parameters**               | _nlist_, _nprobe_                                                                                                              | _m_, _ef_construction_, _ef_search_                                                                                                                                         |
| **Use Cases**                    | When memory is limited, but high query performance and recall are required; well-suited for searches with filtering conditions | When memory is sufficient and the goal is extremely high recall and query performance, but filtering is not needed, or the filtering ratio is low                             |

In real-world applications, it’s very common to include filtering conditions—for example, “only search vectors from a specific user” or “limit results to a certain time range.” Due to differences in their underlying algorithms, IVF generally handles filtered searches more efficiently than HNSW.

The strength of IVF lies in its two-level filtering process. It can first perform a coarse-grained filter at the centroid (cluster) level to quickly narrow down the candidate set, and then conduct fine-grained distance calculations within the selected clusters. This maintains stable and predictable performance, even when a large portion of the data is filtered out.

In contrast, HNSW is based on graph traversal. Because of its structure, it cannot directly leverage filtering conditions during traversal. When the filtering ratio is low, this doesn’t cause major issues. However, when the filtering ratio is high (e.g., more than 90% of data is filtered out), the remaining graph often becomes fragmented, forming many “isolated nodes.” In such cases, the search may degrade into a near full-graph traversal—sometimes even worse than a brute-force search.

In practice, IVF indexes are already powering many high-impact use cases across different domains:

- **E-commerce search:** A user can upload a product image and instantly find visually similar items from millions of listings.

- **Patent retrieval:** Given a short description, the system can locate the most semantically related patents from a massive database—far more efficient than traditional keyword search.

- **RAG knowledge bases:** IVF helps retrieve the most relevant context from millions of tenant documents, ensuring AI models generate more accurate and grounded responses.


## Conclusion

To choose the right index, it all comes down to your specific use case. If you’re working with large-scale datasets or need to support filtered searches, IVF can be the better fit. Compared with graph-based indexes like HNSW, IVF delivers faster index builds, lower memory usage, and a strong balance between speed and accuracy.

[Milvus](https://milvus.io/), the most popular open-source vector database, provides full support for the entire IVF family, including IVF_FLAT, IVF_PQ, and IVF_SQ8. You can easily experiment with these index types and find the setup that best fits your performance and memory needs. For a complete list of indexes Milvus supports, check out this [Milvus Index doc page](https://milvus.io/docs/index-explained.md). 

If you’re building image search, recommendation systems, or RAG knowledge bases, give IVF indexing in Milvus a try—and see how efficient, large-scale vector search feels in action.

How to Choose Between IVF and HNSW for ANN Vector Search

Understanding IVF Vector Index: How It Works and When to Choose It Over HNSW



This post was originally published on [Medium](https://medium.com/opus-engineering/how-opussearch-built-exact-matching-for-enterprise-rag-with-milvus-bm25-aa1098a9888b) and is reposted here with permission.

## The Semantic Search Blind Spot

Picture this: You’re a video editor on deadline. You need clips from “episode 281” of your podcast. You type it into our search. Our AI-powered semantic search, proud of its intelligence, returns clips from 280, 282, and even suggests episode 218 because the numbers are similar, right?

**Wrong**.

When we launched [OpusSearch](https://www.opus.pro/opussearch) for enterprises in January 2025, we thought semantic search would be enough. Natural language queries like “find funny moments about dating” worked beautifully. Our [Milvus](https://milvus.io/)-powered RAG system was crushing it.

**But then reality hit us in the face with user feedback:**

“I just want clips from episode 281. Why is this so hard?”

“When I search ‘That’s what she said,’ I want EXACTLY that phrase, not ‘that’s what he meant.’”

Turns out that video editors and clippers don’t always want AI to be clever. Sometimes they want software to be **straightforward and correct**.

## Why do we care about Search?

We built an [enterprise search function](https://www.opus.pro/opussearch) because we identified that **monetizing** large video catalogs is the key challenge organizations face. Our RAG-powered platform serves as a **growth agent** that enables enterprises to **search, repurpose, and monetize their entire video libraries**. Read about success case stories from **All The Smoke**, **KFC Radio** and **TFTC** [here](https://www.opus.pro/blog/growing-a-new-youtube-channel-in-90-days-without-creating-new-videos).

## Why We Doubled Down on Milvus (Instead of Adding Another Database)

The obvious solution was to add Elasticsearch or MongoDB for exact matching. However, as a startup, maintaining multiple search systems introduces significant operational overhead and complexity.

![](https://assets.zilliz.com/Strong_community_adoption_with_35k_Git_Hub_stars_fbf773dcdb.webp)

Milvus had recently shipped their full-text search feature, and an evaluation with our own dataset **without any tuning** showed compelling advantages:

- **Superior partial matching accuracy**. For example “drinking story” and “jumping high”, other vector DBs returns sometimes “dining story” and “getting high” which alters the meaning.

- Milvus **returns longer, more comprehensive results** than other databases when queries are general, which is naturally more ideal for our use case.

## Architecture from 5000 feet
![](https://assets.zilliz.com/Milvus_is_the_foundational_vector_database_for_our_Enterprise_RAG_architecture_b3c8ebf39c.webp)
## BM25 + Filtering = Exact Match Magic

Milvus’s full-text search isn’t really about exact matching, but it’s about relevance scoring using BM25 ([Best Matching 25](https://en.wikipedia.org/wiki/Okapi_BM25)), which calculates how relevant a document is to your query. It’s great for “find me something close,” but terrible for “find me exactly this.”

We then **combined BM25’s power with Milvus’s TEXT\_MATCH filtering**. Here’s how it works:

1. **Filter first**: TEXT\_MATCH finds documents containing your exact keywords

2. **Rank second**: BM25 sorts those exact matches by relevance

3. **Win**: You get exact matches, ranked intelligently

Think of it as “give me everything with ‘episode 281’, then show me the best ones first.”

## The Code That Made It Work

### Schema Design

**Important**: We disabled stop words entirely, as terms like “The Office” and “Office” represent distinct entities in our content domain.
```
export function getExactMatchFields(): FieldType[] {
 return [
   {
     name: "id",
     data_type: DataType.VarChar,
     is_primary_key: true,
     max_length: 100,
   },
   {
     name: "text",
     data_type: DataType.VarChar,
     max_length: 1000,
     enable_analyzer: true,
     enable_match: true,  // This is the magic flag
     analyzer_params: {
       tokenizer: 'standard',
       filter: [
         'lowercase',
         {
           type: 'stemmer',
           language: 'english',  // "running" matches "run"
         },
         {
           type: 'stop',
           stop_words: [],  // Keep ALL words (even "the", "a")
         },
       ],
     },
   },
   {
     name: "sparse_vector",
     data_type: DataType.SparseFloatVector,
   },
 ]
}
```
### BM25 Function Setup
```
export const FUNCTIONS: FunctionObject[] = [
 {
   name: 'text_bm25_embedding',
   type: FunctionType.BM25,
   input_field_names: ['text'],
   output_field_names: ['sparse_vector'],
   params: {},
 },
]
```
### Index Config

These bm25\_k1 and bm25\_b parameters were tuned against our production dataset for optimal performance.

**bm25\_k1**: Higher values (up to \~2.0) give more weight to repeated term occurrences, while lower values reduce the impact of term frequency after the first few occurrences.

**bm25\_b**: Values closer to 1.0 heavily penalize longer documents, while values closer to 0 ignore document length entirely.
```
index_params: [
 {
   field_name: 'sparse_vector',
   index_type: 'SPARSE_INVERTED_INDEX',
   metric_type: 'BM25',
   params: {
     inverted_index_algo: 'DAAT_MAXSCORE',
     bm25_k1: 1.2,  // How much does term frequency matter?
     bm25_b: 0.75,  // How much does document length matter?
   },
 },
],
```

### The Search Query That Started Working
```
await this.milvusClient.search({
 collection_name: 'my_collection',
 limit: 30,
 output_fields: ['id', 'text'],
 filter: `TEXT_MATCH(text, "episode 281")`,  // Exact match filter
 anns_field: 'sparse_vector',
 data: 'episode 281',  // BM25 ranking query
})
```
For multi-term exact matches:
```
filter: `TEXT_MATCH(text, "foo") and TEXT_MATCH(text, "bar")`
```
## The Mistakes We Made (So You Don’t Have To)

### Dynamic Fields: Critical for Production Flexibility

Initially, we didn’t enable dynamic fields, which was problematic. Schema modifications required dropping and recreating collections in production environments.

```
await this.milvusClient.createCollection({
 collection_name: collectionName,
 fields: fields,
 enable_dynamic_field: true,  // DO THIS
 // ... rest of config
})
```
### Collection Design: Maintain Clear Separation of Concerns

Our architecture uses dedicated collections per feature domain. This modular approach minimizes the impact of schema changes and improves maintainability.

### Memory Usage: Optimize with MMAP

Sparse indexes require significant memory allocation. For large text datasets, we recommend configuring MMAP to utilize disk storage. This approach requires adequate I/O capacity to maintain performance characteristics.

```
// In your Milvus configuration
use_mmap: true
```
## Production Impact and Results

Following the June 2025 deployment of exact match functionality, we observed measurable improvements in user satisfaction metrics and reduced support volume for search-related issues. Our dual-mode approach enables semantic search for exploratory queries while providing precise matching for specific content retrieval.

The key architectural benefit: maintaining a single database system that supports both search paradigms, reducing operational complexity while expanding functionality.

## What’s Next?

We’re experimenting with **hybrid** **queries combining semantic and exact match in a single search**. Imagine: “Find funny clips from episode 281” where “funny” uses semantic search and “episode 281” uses exact match.

The future of search isn’t picking between semantic AI and exact matching. It’s using **both** intelligently in the same system.


How OpusSearch Built Exact Matching for Enterprise RAG with Milvus

How OpusSearch Built Exact Matching for Enterprise RAG with Milvus BM25


Most AI apps rely on a single model for every request. But that approach quickly runs into limits. Large models are powerful yet expensive, even when they're used for simple queries. Smaller models are cheaper and faster but can’t handle complex reasoning. When traffic surges—say your AI app suddenly goes viral with ten million users overnight—the inefficiency of this one-model-for-all setup becomes painfully apparent. Latency spikes, GPU bills explode, and the model that ran fine yesterday starts gasping for air.

And my friend, _you_, the engineer behind this app, have to fix it—fast.

Imagine deploying multiple models of varying sizes and having your system automatically select the best one for each request. Simple prompts go to smaller models; complex queries route to larger ones. That’s the idea behind the [**vLLM Semantic Router**](https://github.com/vllm-project/semantic-router)—a routing mechanism that directs requests based on meaning, not endpoints. It analyzes the semantic content, complexity, and intent of each input to select the most suitable language model, ensuring every query is handled by the model best equipped for it.

To make this even more efficient, the Semantic Router pairs with [**Milvus**](https://milvus.io/), an open-source vector database that serves as a **semantic cache layer**. Before recomputing a response, it checks whether a semantically similar query has already been processed and instantly retrieves the cached result if found. The result: faster responses, lower costs, and a retrieval system that scales intelligently rather than wastefully.

In this post, we’ll dive deeper into how the **vLLM Semantic Router** works, how **Milvus** powers its caching layer, and how this architecture can be applied in real-world AI applications.

## What is a Semantic Router? 

At its core, a **Semantic Router** is a system that decides _which model_ should handle a given request based on its meaning, complexity, and intent. Instead of routing everything to one model, it distributes requests intelligently across multiple models to balance accuracy, latency, and cost.

Architecturally, it’s built on three key layers: **Semantic Routing**, **Mixture of Models (MoM)**, and a **Cache Layer**.

### Semantic Routing Layer 

The **semantic routing layer** is the brain of the system. It analyzes each input—what it’s asking, how complex it is, and what kind of reasoning it requires—to select the model best suited for the job. For example, a simple fact lookup might go to a lightweight model, while a multi-step reasoning query is routed to a larger one. This dynamic routing keeps the system responsive even as traffic and query diversity increase.

![](https://assets.zilliz.com/modern_approach_714403b61c.png)

### The Mixture of Models (MoM) Layer 

The second layer, the **Mixture of Models (MoM)**, integrates multiple models of different sizes and capabilities into one unified system. It’s inspired by the [**Mixture of Experts**](https://zilliz.com/learn/what-is-mixture-of-experts) **(MoE)** architecture, but instead of picking “experts” inside a single large model, it operates across multiple independent models. This design reduces latency, lowers costs, and avoids being locked into any single model provider.

![](https://assets.zilliz.com/MOM_0a3eb61985.png)

### The Cache Layer: Where Milvus Makes the Difference

Finally, the **cache layer**—powered by [Milvus Vector Database](https://milvus.io/)—acts as the system’s memory. Before running a new query, it checks whether a semantically similar request has been processed before. If so, it retrieves the cached result instantly, saving compute time and improving throughput.

Traditional caching systems rely on in-memory key-value stores, matching requests by exact strings or templates. That works fine when queries are repetitive and predictable. But real users rarely type the same thing twice. Once the phrasing changes—even slightly—the cache fails to recognize it as the same intent. Over time, the cache hit rate drops, and performance gains vanish as language naturally drifts.

![](https://assets.zilliz.com/semantic_caching_for_vllm_routing_df889058c9.webp)

To fix this, we need caching that understands _meaning_, not just matching words. That’s where **semantic retrieval** comes in. Instead of comparing strings, it compares embeddings—high-dimensional vector representations that capture semantic similarity. The challenge, though, is scale. Running a brute-force search across millions or billions of vectors on a single machine (with time complexity O(N·d)) is computationally prohibitive. Memory costs explode, horizontal scalability collapses, and the system struggles to handle sudden traffic spikes or long-tail queries.

![](https://assets.zilliz.com/milvus_routing_system_5837b93074.png)

As a distributed vector database purpose-built for large-scale semantic search, **Milvus** brings the horizontal scalability and fault tolerance this cache layer needs. It stores embeddings efficiently across nodes and performs [Approximate Nearest Neighbo](https://zilliz.com/blog/ANN-machine-learning)r (ANN) searches with minimal latency, even at massive scale. With the right similarity thresholds and fallback strategies, Milvus ensures stable, predictable performance—turning the cache layer into a resilient semantic memory for your entire routing system.


## How Developers Are Using Semantic Router + Milvus in Production

The combination of **vLLM Semantic Router** and **Milvus** shines in real-world production environments where speed, cost, and reusability all matter. 

Three common scenarios stand out:


### 1. Customer Service Q\&A

Customer-facing bots handle massive volumes of repetitive queries every day—password resets, account updates, delivery statuses. This domain is both cost- and latency-sensitive, making it ideal for semantic routing. The router sends routine questions to smaller, faster models and escalates complex or ambiguous ones to larger models for deeper reasoning. Meanwhile, Milvus caches previous Q\&A pairs, so when similar queries appear, the system can instantly reuse past answers instead of regenerating them.


### 2. Code Assistance

In developer tools or IDE assistants, many queries overlap—syntax help, API lookups, small debugging hints. By analyzing the semantic structure of each prompt, the router dynamically selects an appropriate model size: lightweight for simple tasks, more capable for multi-step reasoning. Milvus boosts responsiveness further by caching similar coding problems and their solutions, turning prior user interactions into a reusable knowledge base.


### 3. Enterprise Knowledge Base

Enterprise queries tend to repeat over time—policy lookups, compliance references, product FAQs. With Milvus as the semantic cache layer, frequently asked questions and their answers can be stored and retrieved efficiently. This minimizes redundant computation while keeping responses consistent across departments and regions.

Under the hood, the **Semantic Router + Milvus** pipeline is implemented in **Go** and **Rust** for high performance and low latency. Integrated at the gateway layer, it continuously monitors key metrics—like hit rates, routing latency, and model performance—to fine-tune routing strategies in real time.

## How to Quickly Test the Semantic Caching in the Semantic Router

Before deploying semantic caching at scale, it’s useful to validate how it behaves in a controlled setup. In this section, we’ll walk through a quick local test that shows how the Semantic Router uses **Milvus** as its semantic cache. You’ll see how similar queries hit the cache instantly while new or distinct ones trigger model generation—proving the caching logic in action.

### Prerequisites

- Container Environment: Docker + Docker Compose
- Vector Database: Milvus Service
- LLM + Embedding: Project downloaded locally

### 1.Deploy the Milvus Vector Database

Download the deployment files

```
wget https://github.com/Milvus-io/Milvus/releases/download/v2.5.12/Milvus-standalone-docker-compose.yml -O docker-compose.yml
```

Start the Milvus service.

```
docker-compose up -d
```

```
docker-compose ps -a
```

![](https://assets.zilliz.com/Start_the_Milvus_service_211f8b11f1.webp)


### 2. Clone the project

```
git clone https://github.com/vllm-project/semantic-router.git
```

### 3. Download local models

```
cd semantic-router
make download-models
```

![](https://assets.zilliz.com/Download_local_models_6243011fa5.webp)

### 4. Configuration Modifications
Note: Modify the semantic_cache type to milvus

```
vim config.yaml
```

```
semantic_cache:
  enabled: true
  backend_type: "milvus"  # Options: "memory" or "milvus"
  backend_config_path: "config/cache/milvus.yaml"
  similarity_threshold: 0.8
  max_entries: 1000  # Only applies to memory backend
  ttl_seconds: 3600
  eviction_policy: "fifo"
```
Modify the Mmilvus configuration
Note: Fill in the Milvusmilvus service just deployed
```
vim milvus.yaml
```

```
# Milvus connection settings
connection:
  # Milvus server host (change for production deployment)
  host: "192.168.7.xxx"  # For production: use your Milvus cluster endpoint
  # Milvus server port
  port: 19530  # Standard Milvus port
  # Database name (optional, defaults to "default")
  database: "default"
  # Connection timeout in seconds
  timeout: 30
  # Authentication (enable for production)
  auth:
    enabled: false  # Set to true for production
    username: ""    # Your Milvus username
    password: ""    # Your Milvus password
  # TLS/SSL configuration (recommended for production)
  tls:
    enabled: false      # Set to true for secure connections
    cert_file: ""       # Path to client certificate
    key_file: ""        # Path to client private key
    ca_file: ""         # Path to CA certificate
# Collection settings
collection:
  # Name of the collection to store cache entries
  name: "semantic_cache"
  # Description of the collection
  description: "Semantic cache for LLM request-response pairs"
  # Vector field configuration
  vector_field:
    # Name of the vector field
    name: "embedding"
    # Dimension of the embeddings (auto-detected from model at runtime)
    dimension: 384  # This value is ignored - dimension is auto-detected from the embedding model
    # Metric type for similarity calculation
    metric_type: "IP"  # Inner Product (cosine similarity for normalized vectors)
  # Index configuration for the vector field
  index:
    # Index type (HNSW is recommended for most use cases)
    type: "HNSW"
    # Index parameters
    params:
      M: 16              # Number of bi-directional links for each node
      efConstruction: 64  # Search scope during index construction
```

### 5. Start the project
Note: It is recommended to modify some Dockerfile dependencies to domestic sources

```
docker compose --profile testing up --build
```

![](https://assets.zilliz.com/Start_the_project_4e7c2a8332.webp)

### 6. Test Requests
Note: Two requests in total (no cache and cache hit)
First request:
```
echo "=== 第一次请求（无缓存状态） ===" && \
time curl -X POST http://localhost:8801/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer test-token" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What are the main renewable energy sources?"}
    ],
    "temperature": 0.7
  }' | jq .
```
Output:
```
real    0m16.546s
user    0m0.116s
sys     0m0.033s
```
Second request:
```
echo "=== 第二次请求（缓存状态） ===" && \
time curl -X POST http://localhost:8801/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer test-token" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What are the main renewable energy sources?"}
    ],
    "temperature": 0.7
  }' | jq .
```
Output:
```
real    0m2.393s
user    0m0.116s
sys     0m0.021s
```
This test demonstrates Semantic Router's semantic caching in action. By leveraging Milvus as the vector database, it efficiently matches semantically similar queries, improving response times when users ask the same or similar questions.

## Conclusion

As AI workloads grow and cost optimization becomes essential, the combination of vLLM Semantic Router and [Milvus](https://milvus.io/) provides a practical way to scale intelligently. By routing each query to the right model and caching semantically similar results with a distributed vector database, this setup cuts compute overhead while keeping responses fast and consistent across use cases.

In short, you get smarter scaling—less brute force, more brains.

If you’d like to explore this further, join the conversation in our [Milvus Discord](https://discord.com/invite/8uyFbECzPX) or open an issue on[ GitHub](https://github.com/milvus-io/milvus). You can also book a 20-minute[ Milvus Office Hours session](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) for one-on-one guidance, insights, and technical deep dives from the team behind Milvus.


Scale Your AI Apps the Smart Way with vLLM Semantic Router and Milvus

vLLM Semantic Router + Milvus: How Semantic Routing and Caching Build Scalable AI Systems the Smart Way


A Deep Dive into Milvus Analyzer for Full-Text Search


Tokenize Smarter, Retrieve Better: A Deep Dive into Milvus Analyzer for Full-Text Search



Building a robust RAG system usually starts with **document** [**chunking**](https://zilliz.com/learn/guide-to-chunking-strategies-for-rag#Chunking)—splitting large texts into manageable pieces for embedding and retrieval. Common strategies include:

*   **Fixed‑size chunks** (e.g., every 512 tokens)
    
*   **Variable‑size chunks** (e.g., paragraph or sentence boundaries)
    
*   **Sliding windows** (overlapping spans)
    
*   **Recursive chunking** (hierarchical splits)
    
*   **Semantic chunking** (grouping by topic)
    

While these methods have their merits, they often fracture long‑range context. To address this challenge, Jina AI creates a Late Chunking approach: embed the entire document first, then carve out your chunks. 

In this article, we'll explore how Late Chunking works and demonstrate how combining it with [Milvus](https://milvus.io/)—a high-performance open-source vector database built for similarity search—can dramatically improve your RAG pipelines. Whether you're building enterprise knowledge bases, AI-driven customer support, or advanced search applications, this walkthrough will show you how to manage embeddings more effectively at scale.

## What Is Late Chunking?

Traditional chunking methods can break important connections when key information spans multiple chunks—resulting in poor retrieval performance. 

Consider these release notes for Milvus 2.4.13, split into two chunks like below: 

![](https://assets.zilliz.com/Figure1_Chunking_Milvus2_4_13_Release_Note_fe7fbdb833.png)


_Figure 1. Chunking Milvus 2.4.13 Release Note_


If you query, “What are the new features in Milvus 2.4.13?”, a standard embedding model may fail to link “Milvus 2.4.13” (in Chunk 1) with its features (in Chunk 2). The result? Weaker vectors and lower retrieval accuracy.

Heuristic fixes—such as sliding windows, overlapping contexts, and repeated scans—provide partial relief but no guarantees.

**Traditional chunking** follows this pipeline:

1.  **Pre‑chunk** text (by sentences, paragraphs, or max token length).
    
2.  **Embed** each chunk separately.
    
3.  **Aggregate** token embeddings (e.g., via average pooling) into a single chunk vector.
    

**Late Chunking** flips the pipeline:

1.  **Embed first**: Run a long‑context transformer over the full document, generating rich token embeddings that capture global context.
    
2.  **Chunk later**: Average‑pool contiguous spans of those token embeddings to form your final chunk vectors.
    
![](https://assets.zilliz.com/Figure2_Naive_Chunkingvs_Late_Chunking_a94d30b6ea.png)


_Figure 2. Naive Chunking vs. Late Chunking (_[_Source_](https://jina.ai/news/late-chunking-in-long-context-embedding-models/)_)_



By preserving full‑document context in every chunk, Late Chunking yields:

*   **Higher retrieval accuracy**—each chunk is contextually aware.
    
*   **Fewer chunks**—you send more focused text to your LLM, cutting costs and latency.
    

Many long‑context models like jina-embeddings-v2-base-en can process up to 8,192 tokens—equivalent to about a 20-minute read (roughly 5,000 words)—making Late Chunking practical for most real‑world documents.

Now that we understand the “what” and “why” behind Late Chunking, let’s dive into the “how”. In the next section, we’ll guide you through a hands‑on implementation of the Late Chunking pipeline, benchmark its performance against traditional chunking, and validate its real‑world impact using Milvus. This practical walkthrough will bridge theory and practice, showing exactly how to integrate Late Chunking into your RAG workflows.

## Testing Late Chunking

### Basic Implementation

Below are the core functions for Late Chunking. We’ve added clear docstrings to guide you through each step. The function `sentence_chunker` splits the original document into paragraph‑based chunks, returning both the chunk contents and the chunk annotation information `span_annotations` (i.e., the start and end indices of each chunk).

```
def sentence_chunker(document, batch_size=10000):
    nlp = spacy.blank("en")
    nlp.add_pipe("sentencizer", config={"punct_chars": None})
    doc = nlp(document)

    docs = []
    for i in range(0, len(document), batch_size):
        batch = document[i : i + batch_size]
        docs.append(nlp(batch))

    doc = Doc.from_docs(docs)

    span_annotations = []
    chunks = []
    for i, sent in enumerate(doc.sents):
        span_annotations.append((sent.start, sent.end))
        chunks.append(sent.text)

    return chunks, span_annotations
```


The function `document_to_token_embeddings` uses the jinaai/jina-embeddings-v2-base-en model and its tokenizer to produce embeddings for the entire document.

```
def document_to_token_embeddings(model, tokenizer, document, batch_size=4096):
    tokenized_document = tokenizer(document, return_tensors="pt")
    tokens = tokenized_document.tokens()

    outputs = []
    for i in range(0, len(tokens), batch_size):
        
        start = i
        end   = min(i + batch_size, len(tokens))

        batch_inputs = {k: v[:, start:end] for k, v in tokenized_document.items()}

        with torch.no_grad():
            model_output = model(**batch_inputs)

        outputs.append(model_output.last_hidden_state)

    model_output = torch.cat(outputs, dim=1)
    return model_output
```


The function `late_chunking` takes the document’s token embeddings and the original chunk annotation information `span_annotations`, and then produces the final chunk embeddings.

```
def late_chunking(token_embeddings, span_annotation, max_length=None):
    outputs = []
    for embeddings, annotations in zip(token_embeddings, span_annotation):
        if (
            max_length is not None
        ):
            annotations = [
                (start, min(end, max_length - 1))
                for (start, end) in annotations
                if start < (max_length - 1)
            ]
        pooled_embeddings = []
        for start, end in annotations:
            if (end - start) >= 1:
                pooled_embeddings.append(
                    embeddings[start:end].sum(dim=0) / (end - start)
                )
                    
        pooled_embeddings = [
            embedding.detach().cpu().numpy() for embedding in pooled_embeddings
        ]
        outputs.append(pooled_embeddings)

    return outputs
```



For example, chunking with jinaai/jina-embeddings-v2-base-en:

```
tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True)
model     = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True)

# First chunk the text as normal, to obtain the beginning and end points of the chunks.
chunks, span_annotations = sentence_chunker(document)
# Then embed the full document.
token_embeddings = document_to_token_embeddings(model, tokenizer, document)
# Then perform the late chunking
chunk_embeddings = late_chunking(token_embeddings, [span_annotations])[0]
```


_Tip:_ Wrapping your pipeline in functions makes it easy to swap in other long‑context models or chunking strategies.

### Comparison with Traditional Embedding Methods

To further demonstrate the advantages of Late Chunking, we also compared it to traditional embedding approaches, using a set of sample documents and queries.

And let’s revisit our Milvus 2.4.13 release note example:

```
Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection. This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery. Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency. We highly recommend upgrading to this release for better performance and stability.
```


We measure [cosine similarity](https://zilliz.com/blog/similarity-metrics-for-vector-search#Cosine-Similarity) between the query embedding ("milvus 2.4.13") and each chunk:

```
cos_sim = lambda x, y: np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))

milvus_embedding = model.encode('milvus 2.4.13')

for chunk, late_chunking_embedding, traditional_embedding in zip(chunks, chunk_embeddings, embeddings_traditional_chunking):
    print(f'similarity_late_chunking("milvus 2.4.13", "{chunk}")')
    print('late_chunking: ', cos_sim(milvus_embedding, late_chunking_embedding))
    print(f'similarity_traditional("milvus 2.4.13", "{chunk}")')
    print('traditional_chunking: ', cos_sim(milvus_embedding, traditional_embeddings))
```


Late Chunking consistently outperformed traditional chunking, yielding higher cosine similarities across every chunk. This confirms that embedding the full document first preserves global context more effectively.

```
similarity_late_chunking("milvus 2.4.13", "Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection.")
late_chunking: 0.8785206
similarity_traditional("milvus 2.4.13", "Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection.")
traditional_chunking: 0.8354263

similarity_late_chunking("milvus 2.4.13", "This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery.")
late_chunking: 0.84828955
similarity_traditional("milvus 2.4.13", "This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery.")
traditional_chunking: 0.7222632

similarity_late_chunking("milvus 2.4.13", "Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency.")
late_chunking: 0.84942204
similarity_traditional("milvus 2.4.13", "Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency.")
traditional_chunking: 0.6907381

similarity_late_chunking("milvus 2.4.13", "We highly recommend upgrading to this release for better performance and stability.")
late_chunking: 0.85431844
similarity_traditional("milvus 2.4.13", "We highly recommend upgrading to this release for better performance and stability.")
traditional_chunking: 0.71859795
```


We can see that embedding the full paragraph first ensures each chunk carries the “`Milvus 2.4.13`” context—boosting similarity scores and retrieval quality.

### **Testing Late Chunking in Milvus**

Once chunk embeddings are generated, we can store them in Milvus and perform queries. The following code inserts chunk vectors into the collection.

#### **Importing Embeddings into Milvus**

```
batch_data=[]
for i in range(len(chunks)):
    data = {
            "content": chunks[i],
            "embedding": chunk_embeddings[i].tolist(),
        }

    batch_data.append(data)

res = client.insert(
    collection_name=collection,
    data=batch_data,
)
```



#### Querying and Validation

To validate the accuracy of Milvus queries, we compare its retrieval results to brute-force cosine similarity scores calculated manually. If both methods return consistent top‑k results, we can be confident that Milvus’s search accuracy is reliable.

We compare Milvus’s native search with a brute‑force cosine similarity scan:

```
def late_chunking_query_by_milvus(query, top_k = 3):
    query_vector = model(**tokenizer(query, return_tensors="pt")).last_hidden_state.mean(1).detach().cpu().numpy().flatten()

    res = client.search(
                collection_name=collection,
                data=[query_vector.tolist()],
                limit=top_k,
                output_fields=["id", "content"],
            )

    return [item.get("entity").get("content") for items in res for item in items]

def late_chunking_query_by_cosine_sim(query, k = 3):
    cos_sim = lambda x, y: np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))
    query_vector = model(**tokenizer(query, return_tensors="pt")).last_hidden_state.mean(1).detach().cpu().numpy().flatten()

    results = np.empty(len(chunk_embeddings))
    for i, (chunk, embedding) in enumerate(zip(chunks, chunk_embeddings)):
        results[i] = cos_sim(query_vector, embedding)

    results_order = results.argsort()[::-1]
    return np.array(chunks)[results_order].tolist()[:k]
```


This confirms Milvus returns the same top‑k chunks as a manual cosine‑sim scan.

```
> late_chunking_query_by_milvus("What are new features in milvus 2.4.13", 3)

['\n\n### Features\n\n- Dynamic replica adjustment for loaded collections ([#36417](https://github.com/milvus-io/milvus/pull/36417))\n- Sparse vector MMAP in growing segment types ([#36565](https://github.com/milvus-io/milvus/pull/36565))...
```

```
> late_chunking_query_by_cosine_sim("What are new features in milvus 2.4.13", 3)

['\n\n### Features\n\n- Dynamic replica adjustment for loaded collections ([#36417](https://github.com/milvus-io/milvus/pull/36417))\n- Sparse vector MMAP in growing segment types (#36565)...
```


So both methods yield the same top-3 chunks, confirming Milvus’s accuracy.

## Conclusion


In this article, we took a deep dive into the mechanics and benefits of Late Chunking. We began by identifying the shortcomings of traditional chunking approaches, particularly when handling long documents where preserving context is crucial. We introduced the concept of Late Chunking—embedding the entire document before slicing it into meaningful chunks—and showed how this preserves global context, leading to improved semantic similarity and retrieval accuracy.

We then walked through a hands-on implementation using Jina AI’s jina-embeddings-v2-base-en model and evaluated its performance compared to traditional methods. Finally, we demonstrated how to integrate the chunk embeddings into Milvus for scalable and accurate vector search.

Late Chunking offers a **context‑first** approach to embedding—perfect for long, complex documents where context matters most. By embedding the entire text upfront and slicing later, you gain:

*   🔍 **Sharper retrieval accuracy**
    
*   ⚡ **Lean, focused LLM prompts**
    
*   🛠️ **Simple integration** with any long‑context model


We Extracted OpenClaw’s Memory System and Open-Sourced It (memsearch)

7 Best Open-Source Tools for Claude Code Context Management

Claude Context: Reduce Claude Code Token Usage with Milvus-Powered Code Retrieval

How RoboBrain Builds Long-Term Robot Memory with Milvus

Be the first to know

7 Best Open-Source Tools for Claude Code Context Management

Claude Context: Reduce Claude Code Token Usage with Milvus-Powered Code Retrieval

How RoboBrain Builds Long-Term Robot Memory with Milvus

DeepSeek V4 vs GPT-5.5 vs Qwen3.6: Which Model Should You Use?

How to Fix Hermes Agent's Learning Loop with Milvus 2.6 Hybrid Search

How to Add Long-Term Memory to Anthropic's Managed Agents with Milvus

Interview with RaBitQ Authors: The TurboQuant Dispute and Why the Storage Selloff Was a False Alarm

We Built Graph RAG Without the Graph Database

Harness Engineering: The Execution Layer AI Agents Actually Need

Ask me anything about Milvus!

Zilliz Cloud

Hassle-free and 10x faster than Milvus

Most Contributing Authors

Stefan Webb

David Wang

Jiang Chen