RAG vs MCP vs API: what each one solves and when to use which

"Should we use RAG or MCP?" is one of the most common questions we get from teams starting with AI agents, usually followed by "and why not just call the API?". The three get compared as if they were competing options. They are not. They solve different problems, and most real systems use at least two of them.

This guide explains what each one actually is, gives a decision table, and shows the pattern that ties them together: an MCP server that exposes retrieval. If MCP itself is new to you, start with what an MCP server is.

The short version

• RAG answers the question "which text should the model read before it answers?"
• An API answers "how does one program talk to another?"
• MCP answers "how does an AI application discover and use tools and data it was not hardcoded for?"

RAG is a technique. An API is a contract. MCP is a protocol for exposing APIs and data to models. You pick a technique for the knowledge problem and a protocol for the integration problem.

What RAG solves

Retrieval-augmented generation was named in a 2020 paper by Lewis et al. at Facebook AI Research. The idea: a model's built-in knowledge is frozen at training time and does not include your company's documents, so before generating an answer you search a document store, take the most relevant passages and put them in the prompt.

A RAG pipeline has a few moving parts:

1. Ingestion. Split documents into chunks, attach metadata (source, date, owner, access rights).
2. Indexing. Embeddings for vector search, often combined with keyword search (hybrid).
3. Retrieval and ranking. Turn the question into a query, fetch candidates, rerank.
4. Generation. The model answers from the retrieved passages and cites them.

RAG is the right tool when the knowledge is large, unstructured and mostly read-only: policies, contracts, product documentation, support articles, case files. Its value is grounding: answers trace back to a source someone can check.

Its weak spots are well known. Retrieval quality decides answer quality, so bad chunking or a poor index gives confident wrong answers. The index goes stale unless ingestion keeps up. And access control has to be carried into the index, or a user can retrieve a document they could never open directly.

What a plain API solves

An API is a contract between programs: an endpoint, a request format, a response format, an auth scheme. A developer reads the docs and writes code that calls it.

curl -s https://api.example.com/v1/orders/48213 \
  -H "Authorization: Bearer $TOKEN"

Nothing here involves a model. The code decides exactly which call to make, with which arguments, every time. That is the strength: deterministic, testable, cheap and fast. It is also the limit. Each integration is custom code, and if you want a model to use it, you have to write the glue that turns the API into something the model can call: a tool definition, argument validation, error handling, auth.

What MCP solves

The Model Context Protocol, introduced by Anthropic and since donated to the Agentic AI Foundation under the Linux Foundation, standardises that glue. An MCP server exposes three kinds of things, as the specification defines them:

• Tools: functions the model can execute. The spec calls them model-controlled: the model discovers them and decides when to call them.
• Resources: data such as files or records, identified by URI. These are application-driven: the host app decides how to bring them into context.
• Prompts: templated messages and workflows for users.

Messages are JSON-RPC 2.0, carried over stdio for local servers or Streamable HTTP for remote ones. The key difference from a plain API is that the server describes itself in a form a model can read. A client asks for the tool list and gets back names, descriptions and a JSON Schema per tool:

{
  "name": "search_policies",
  "description": "Search internal HR and finance policies. Returns the most relevant passages with document title and URL.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": { "type": "string", "description": "What the user is asking about" },
      "top_k": { "type": "integer", "description": "Number of passages to return" }
    },
    "required": ["query"]
  }
}

Write the server once and it works in any MCP client: Claude, ChatGPT, Cursor, Copilot, Claude Code, your own agent. That is the "MCP vs API" answer in one line: MCP is not a replacement for your API, it is a standard wrapper that lets models use it without per-client integration code. Most MCP servers are thin layers over an existing REST API.

RAG vs MCP vs API: decision table

Two rules of thumb sit behind the table. If the data is structured and live, give the model a tool that queries the source instead of indexing it. If it is unstructured and large, you need retrieval, whatever you deliver it with.

How they combine: an MCP server that exposes retrieval

The most useful pattern is not "RAG or MCP" but RAG behind MCP. The retrieval pipeline does what it is good at, and MCP makes it a tool any agent can call when it decides it needs knowledge.

The agent sees one tool, for example search_policies above. A call looks like this:

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "tools/call",
  "params": {
    "name": "search_policies",
    "arguments": { "query": "parental leave rules for part-time staff", "top_k": 5 }
  }
}

Behind it, the server runs hybrid search, reranks, filters by the caller's permissions, and returns passages with titles and links. The model then answers and cites them, and in the same session it can call other tools: look up the employee's contract type in the HR system, or draft a reply in the ticketing tool.

This is not hypothetical. Microsoft's Azure AI Search exposes each knowledge base as an MCP endpoint with a knowledge_base_retrieve tool, which plans subqueries, runs keyword, vector or hybrid search and returns results with source references. Microsoft's docs list Foundry Agent Service, GitHub Copilot, Claude and Cursor as clients that can call it. Several of the vendor servers in our best MCP servers list, such as Notion's, also include a search tool, which is a lightweight form of the same idea over a single system.

Design points that matter when you build one:

• Return passages and sources, not whole documents. Context is limited and citations are the point.
• Enforce access per user. The tool should filter by the caller's identity, not by a shared service account that sees everything.
• Keep retrieval read-only. Separate search tools from write tools so approval policies stay simple.
• Treat retrieved text as untrusted. A document can contain a prompt injection. See MCP security best practices.
• Write a good tool description. The model decides when to search based on it. Say what the corpus covers and what it does not.

Where teams go wrong

• Indexing data that should be queried. Embedding a CRM export gives the agent yesterday's numbers. A tool over the live API is simpler and correct.
• Building RAG for a few pages. If the whole corpus fits comfortably in the prompt, retrieval is overhead.
• Exposing every API endpoint as a tool. Dozens of near-identical tools make models choose badly. Fewer, task-shaped tools work better.
• Skipping the API when no model is involved. Not every integration needs an agent. A cron job calling an API is still the right answer for a fixed flow.

For how retrieval and tools fit into a full agent loop, see how to build AI agents.

Governance: where RAG and MCP meet company data

Once retrieval and tools are exposed to many users and several models, the hard questions are no longer technical. Which models may see which knowledge bases? Where is the data processed? Who called which tool, with what result? Those are policy questions, and they belong in one place rather than in each client's settings, which is the job of an MCP gateway.

At Walma we build custom RAG pipelines and agents for exactly this setup, and deliver them as MCP tools through Walma AI Hub, which runs in the customer's own Azure tenant in an EU region with one policy and one log across models and connections. A support knowledge base exposed this way is the core of our AI for customer service offering. If you have a document-heavy use case, see our custom solutions, or browse the full MCP guide for the rest of the cluster.