days, it’s all about brokers, which I’m all for, anbeyond fundamental vector search by giving LLMs entry to a variety of instruments:
- Net search
- Varied API calls
- Querying totally different databases
Whereas there’s a surge in new MCP servers being developed, there’s surprisingly little analysis taking place. Certain, you may hook an LLM with numerous totally different instruments, however do you actually know the way it’s going to behave? That’s why I’m planning a sequence of weblog posts centered on evaluating each off-the-shelf and customized graph MCP servers, particularly those who retrieve data from Neo4j.
Mannequin Context Protocol (MCP) is Anthropic’s open customary that features like “a USB-C port for AI purposes,” standardizing how AI methods hook up with exterior knowledge sources by way of light-weight servers that expose particular capabilities to purchasers. The important thing perception is reusability. As an alternative of customized integrations for each knowledge supply, builders construct reusable MCP servers as soon as and share them throughout a number of AI purposes.
An MCP server implements the Mannequin Context Protocol, exposing instruments and knowledge to an AI consumer by way of structured JSON-RPC calls. It handles requests from the consumer and executes them in opposition to native or distant APIs, returning outcomes to counterpoint the AI’s context.
To guage MCP servers and their retrieval strategies, step one is to generate an analysis dataset, one thing we’ll use an LLM to assist with. Within the second stage, we’ll take an off-the-shelf mcp-neo4j-cypher server and check it in opposition to the benchmark dataset we created.
The objective for now’s to ascertain a strong dataset and framework so we will constantly examine totally different retrievers all through the sequence.
Code is on the market on GitHub.
Analysis dataset
Final 12 months, Neo4j launched the Text2Cypher (2024) Dataset, which was designed round a single-step method to Cypher era. In single-step Cypher era, the system receives a pure language query and should produce one full Cypher question that straight solutions that query, primarily a one-shot translation from textual content to database question.
Nonetheless, this method doesn’t replicate how brokers truly work with graph databases in observe. Brokers function by way of multi-step reasoning: they’ll execute a number of instruments iteratively, generate a number of Cypher statements in sequence, analyze intermediate outcomes, and mix findings from totally different queries to construct as much as a closing reply. This iterative, exploratory method represents a basically totally different paradigm from the prescribed single-step mannequin.
The present benchmark dataset fails to seize this distinction of how MCP servers truly get utilized in agentic workflows. The benchmark wants updating to judge multi-step reasoning capabilities somewhat than simply single-shot text2cypher translation. This may higher replicate how brokers navigate complicated data retrieval duties that require breaking down issues, exploring knowledge relationships, and synthesizing outcomes throughout a number of database interactions.
Analysis metrics
An important shift when shifting from single-step text2cypher analysis to an agentic method lies in how we measure accuracy.
In conventional text2query duties like text2cypher, analysis usually includes evaluating the database response on to a predefined floor fact, typically checking for exact matches or equivalence.
Nonetheless, agentic approaches introduce a key change. The agent could carry out a number of retrieval steps, select totally different question paths, and even rephrase the unique intent alongside the way in which. Because of this, there could also be no single appropriate question. As an alternative, we shift our focus to evaluating the ultimate reply generated by the agent, whatever the intermediate queries it used to reach there.
To evaluate this, we use an LLM-as-a-judge setup, evaluating the agent’s closing reply in opposition to the anticipated reply. This lets us consider the semantic high quality and usefulness of the output somewhat than the inner mechanics or particular question outcomes.
End result Granularity and Agent Habits
One other vital consideration in agentic analysis is the quantity of knowledge returned from the database. In conventional text2cypher duties, it’s widespread to permit and even anticipate massive question outcomes, for the reason that objective is to check whether or not the right knowledge is retrieved. Nonetheless, this method doesn’t translate effectively to evaluating agentic workflows.
In an agentic setting, we’re not simply testing whether or not the agent can entry the right knowledge, however whether or not it will possibly generate a concise, correct closing reply. If the database returns an excessive amount of data, the analysis turns into entangled with different variables, such because the agent’s skill to summarize or navigate massive outputs, somewhat than specializing in whether or not it understood the person’s intent and retrieved the right data.
Introducing Actual-World Noise
To additional align the benchmark with real-world agentic utilization, we additionally introduce managed noise into the analysis prompts.
This consists of parts resembling:
- Typographical errors in named entities (e.g., “Andrwe Carnegie” as a substitute of “Andrew Carnegie”),
- Colloquial phrasing or casual language (e.g., “present me what’s up with Tesla’s board” as a substitute of “checklist members of Tesla’s board of administrators”),
- Overly broad or under-specified intents that require follow-up reasoning or clarification.
These variations replicate how customers truly work together with brokers in observe. In actual deployments, brokers should deal with messy inputs, incomplete formulations, and conversational shorthand, that are situations not often captured by clear, canonical benchmarks.
To raised replicate these insights round evaluating agentic approaches, I’ve created a new benchmark using Claude 4.0. Not like conventional benchmarks that concentrate on Cypher question correctness, this one is designed to evaluate the standard of the closing solutions produced by multi-step brokers
Databases
To make sure quite a lot of evaluations, we use a few totally different databases which can be accessible on the Neo4j demo server. Examples embrace:
MCP-Neo4j-Cypher server
mcp-neo4j-cypher is a ready-to-use MCP instrument interface that enables brokers to work together with Neo4j by way of pure language. It helps three core features: viewing the graph schema, operating Cypher queries to learn knowledge, and executing write operations to replace the database. Outcomes are returned in a clear, structured format that brokers can simply perceive and use.
It really works out of the field with any framework that helps MCP servers, making it easy to plug into present agent setups with out further integration work. Whether or not you’re constructing a chatbot, knowledge assistant, or customized workflow, this instrument lets your agent safely and intelligently work with graph knowledge.
Benchmark
Lastly, let’s run the benchmark analysis.
We used LangChain to host the agent and join it to the mcp-neo4j-cypher server, which is the one instrument offered to the agent. This setup makes the analysis easy and life like: the agent should rely totally on pure language interplay with the MCP interface to retrieve and manipulate graph knowledge.
For the analysis, we used Claude 3.7 Sonnet because the agent and GPT-4o Mini because the decide.
The benchmark dataset consists of roughly 200 pure language question-answer pairs, categorized by variety of hops (1-hop, 2-hop, and so on.) and whether or not the queries include distracting or noisy data. This construction helps assess the agent’s reasoning accuracy and robustness in each clear and noisy contexts. The analysis code is on the market on GitHub.
Let’s study the outcomes collectively.
The analysis exhibits that an agent utilizing solely the mcp-neo4j-cypher interface can successfully reply complicated pure language questions over graph knowledge. Throughout a benchmark of round 200 questions, the agent achieved a median rating of 0.71, with efficiency dropping as query complexity elevated. The presence of noise within the enter considerably diminished accuracy, revealing the agent’s sensitivity to typos in named entities and such.
On the instrument utilization facet, the agent averaged 3.6 instrument calls per query. That is in step with the present requirement to make no less than one name to fetch the schema and one other to execute the principle Cypher question. Most queries fell inside a 2–4 name vary, exhibiting the agent’s skill to motive and act effectively. Notably, a small variety of questions had been answered with only one and even zero instrument calls, anomalies that will recommend early stopping, incorrect planning, or agent bugs, and are value additional evaluation. Wanting forward, instrument depend could possibly be diminished additional if schema entry is embedded straight by way of MCP sources, eliminating the necessity for an express schema fetch step.
The true worth of getting a benchmark is that it opens the door to systematic iteration. As soon as baseline efficiency is established, you can begin tweaking parameters, observing their influence, and making focused enhancements. For example, if agent execution is dear, you may wish to check whether or not capping the variety of allowed steps to 10 utilizing a LangGraph recursion restrict has a measurable impact on accuracy. With the benchmark in place, these trade-offs between efficiency and effectivity could be explored quantitatively somewhat than guessed.
With a 10-step restrict in place, efficiency dropped noticeably. The imply analysis rating fell to 0.535. Accuracy decreased sharply on extra complicated (3-hop+) questions, suggesting the step restrict lower off deeper reasoning chains. Noise continued to degrade efficiency, with noisy questions averaging decrease scores than clear ones.
Abstract
We’re dwelling in an thrilling second for AI, with the rise of autonomous brokers and rising requirements like MCP dramatically increasing what LLMs can do, particularly relating to structured, multi-step duties. However whereas the capabilities are rising quick, strong analysis continues to be lagging behind. That’s the place this GRAPE mission is available in.
The objective is to construct a sensible, evolving benchmark for graph-based query answering utilizing the MCP interface. Over time, I plan to refine the dataset, experiment with totally different retrieval methods, and discover the way to lengthen or adapt the Cypher MCP for higher accuracy. There’s nonetheless plenty of work forward from cleansing knowledge, bettering retrieval to tightening analysis. Nonetheless, having a transparent benchmark means we will monitor progress meaningfully, check concepts systematically, and push the boundaries of what these brokers can reliably do.
