Genkidama · details

The experiment, as it stands.

Everything on this page is in progress. It exists so the experimental work has a home while it is being debugged and extended; nothing here is a result yet, and no number is shown that has not been measured.

in progress · no results published
01Experimentwhat is being tested

Same prompts. Same assets. Two routes.

A natively multimodal model reads binary inputs directly. A text-only model reaches the same modalities through Eden AI's MCP specialists. Both see identical prompts and assets, with the MCP catalogue available to both. The experiment records what each capability path costs in latency, tokens, tool traffic and money, and how the answers compare. It does not declare a winner.

02Setuptwo routes to one answer
route a

Native multimodal

  • The model ingests the binary asset directly.
  • No tool traffic; latency is model latency.
  • Capability bounded by the model’s training.
route b

Text model + MCP specialists

  • The model describes what it needs; the gateway routes it.
  • Latency and cost split between model and specialist.
  • Capability bounded by the catalogue, not the model.
03Datasetnot yet selected

A performance dataset is still to be chosen.

  1. 01Identify a benchmark that exercises non-native modalities.
  2. 02Sample it.
  3. 03Define the evaluation methodology.
  4. 04Run the models on the sample.
  5. 05Analyse the results.
04Models2 configurations · more to come
modelnative inputscapability path
Gemini Protext · image · audio · video · pdfnative multimodal
DeepSeek Protexttext + MCP specialists
05MCP pathmodel → mcp → specialist → model

The text-only model emits a tool call; Eden AI's MCP gateway routes it to a specialist and returns the result as text. The catalogue, 39 tools at the time of writing, is discovered at runtime. Per-scenario tool paths are recorded as they are run.MCP server documentation ↗

tool path · Receipt totalreceipt-*.jpg
Gemini Pronative
  1. Geminimodel
  2. native inputbinary input
  3. answeranswer
mcp calls —failures —latency —
DeepSeek Protext + mcp
  1. DeepSeekmodel
  2. MCPgateway
  3. eden.ocr.extractspecialist
  4. DeepSeekmodel
  5. answeranswer
mcp calls —failures —latency —
06Native pathmodel → model

The multimodal model receives the binary asset in its context and answers without tool traffic. Its latency is all model latency and its cost is a single line. This is the reference the MCP path is measured against, not a baseline it must beat.

07Results no results yet
scenario-level comparison placeholder · no results yet
metric
Gemini Pronative multimodal
DeepSeek Protext + mcp specialists
Final-answer agreementDo both models land on the same final answer?
Ground-truth agreementAgreement with the annotated reference answer.
MCP callsTool calls routed through the MCP server.
Tool failuresCalls that returned an error.
RecoveryFailures the model recovered from.
LatencyEnd-to-end wall clock.
Model latencyTime spent inside the model.
Tool latencyTime spent inside specialist tools.
TokensTotal tokens consumed.
Gateway costModel + routing cost.
Expert-tool costSpecialist tool cost.
scenario explorer6 scenarios · identical prompts & assets
prompt

“What is the total amount on this receipt?”

assets

receipt-*.jpg

metricGeminiDeepSeek
Final-answer agreement
Ground-truth agreement
MCP calls
Tool failures
Latency
Gateway cost
08Latencyrecorded model time · recorded tool time · total wall-clock

Three recorded quantities per scenario: model time, MCP/tool time and total wall-clock latency. Round overhead is recorded separately, so the first two do not sum to the third and are never presented as if they did.

latency by scenario
End-to-end latency per scenarioPlaceholder chart; no measurements yet.·····ms (awaiting data)Receipt totals01 · ocrChart readings02 · imageMeeting transcripts03 · speechInvoice fieldss04 · documentsVideo scenes05 · videoLive pages06 · web

Placeholder geometry. Bars carry no values until the benchmark dataset is connected.

09Costtwo lines, kept separate

Gateway cost (model and routing) and expert-tool cost (the specialist) are separate measurements and are never summed.

gateway cost
awaiting data
per scenario · model + routing
expert-tool cost
awaiting data
per scenario · specialist
10Tool usagewhat the model asked for

Per scenario: how many MCP calls the text-only model made, how many failed, and how many failures it recovered from.

mcp calls
awaiting data
per scenario
tool failures
awaiting data
calls that returned an error
recovered
awaiting data
failures the model recovered from
11Analysispending

Written once the measurements above exist. The framing is fixed already: native capability versus MCP-mediated capability, what each path costs and where each is appropriate, not which model wins.

12Limitationsknown, today
  1. 01The harness is still being debugged; recorded runs are not yet trusted enough to publish.
  2. 02Two model configurations so far; more comparisons are planned.
  3. 03No performance dataset selected yet, so no task-quality measurement.
  4. 04Scenario shells are hand-picked, not sampled from a benchmark.
Back to the bridgeBack to the core