Category: Uncategorized

The Model Context Protocol is a communication protocol and a technical standard. It was developed by Anthropic, published on 25 November 2024 under the MIT License, and it is built on JSON-RPC. Its documentation is at modelcontextprotocol.io. This post is about what it does, and why the galaxy exposes itself through it rather than through an interface of our own design.

The problem it addresses

A software agent is a type of tool — a subclass of computer program, of tool, and of artificially intelligent entity. Give one a large language model and it can decide what to do. It still cannot do anything without something to call.

Before a shared protocol, every pairing of a model and a capability was bespoke. Each tool defined its own shape, each host wrote its own adapter, and none of that work transferred. The cost was quadratic in a way nobody chose: with a dozen tools and a handful of hosts, the integrations outnumbered the things being integrated.

The Model Context Protocol replaces that with one interface. A server describes what it offers; a client discovers it and calls it. The model does not learn a new convention per tool, because there is only one.

What actually happens when an agent calls a tool

Being built on JSON-RPC matters more than it sounds. JSON-RPC is deliberately small: a request names a method and carries parameters, a response carries a result or an error, and that is close to the whole specification. The protocol inherits that plainness, so the interesting part is not the transport.

The interesting part is discovery. A client asks a server what it can do, and the server answers with a list of tools, each with a name, a description and a schema for its arguments. The model reads those descriptions and chooses. Then it sends a call, and receives a result.

The consequence people underestimate: the tool description is a prompt. It is the only thing the model sees when deciding whether this tool is the right one. A description that is vague or that describes the implementation rather than the situation the tool is for produces a model that calls the wrong tool confidently, and no amount of correctness in the implementation repairs that.

The same applies to what a tool returns. A large language model uses prompts and the transformer architecture; it has no channel for a result except the text it gets back. A tool that returns a bare success is asking the model to invent the details, and it will.

Why the galaxy is built this way

The 4ort properties are separate systems — search, a knowledge graph, personal memory, simulation, inference routing, hosting. Each is useful alone. The reason an agent can treat them as one nervous system is that each speaks the same protocol, so adding a property costs one server rather than one integration per client.

That is the honest argument for adopting a standard rather than designing an interface: not that the standard is elegant, but that an interface only we implement is one an agent has to be taught about specially. The Model Context Protocol was published openly under the MIT License, which is what makes it worth building against — the licence is the reason the choice is not a bet on somebody else’s roadmap.

What it is not

It is a communication protocol. It is not a permission model, not an identity system, and not a guarantee that a tool does what it says. Those remain the host’s problem, and they get harder rather than easier when a model can reach many tools through one interface: the protocol’s whole value is making capabilities uniformly reachable, which is also the property you have to bound.

Nor does it make a model good at using tools. It makes the calling mechanism uniform. Whether the right tool gets called still comes down to descriptions and schemas, which is to say it comes down to writing — which is why the tool descriptions in our servers are drafted with more care than most of our prose.

The summary

The Model Context Protocol is a small, open, JSON-RPC-based standard from Anthropic, published in November 2024, for letting a software agent discover and call tools. Its value is that it is shared: one interface instead of one per pairing. Everything hard about agent tooling — naming, describing, bounding, returning enough for the model to be honest about what happened — is still hard. The protocol just stops it from being hard a different way for every tool.

Structured data is a data type — a subclass of data whose defining quality is structure, and whose opposite is unstructured data. Making a page machine-readable means moving claims out of the second category and into the first. This is what that actually involves, in the order it is worth doing.

First, know what you are aiming at

Structured data uses a data model, and it is recorded as a facet of the Semantic Web — the information system credited to Tim Berners-Lee, classed as a subclass of the World Wide Web, semantic network and semantic integration, and whose parts include linked data. That lineage explains why the vocabularies feel heavier than a page author expects: they were designed so that statements could be joined across sites, not merely parsed within one.

It is also worth naming the third category the graph is careful about. Structured data is different from semi-structured data. HTML is the everyday example of the latter: it has real structure, and that structure describes presentation rather than meaning. A machine can tell an <h2> from a <p>. It cannot tell a price from a part number, because the markup never said.

Second, use the vocabulary that already exists

Schema.org is a technical standard, an ontology and an organization. It was created on 2 June 2011 by Yandex, Microsoft, Google and Yahoo, its parent organisation is the World Wide Web Consortium, it is published under Creative Commons Attribution-ShareAlike 3.0 Unported, and it has moved through versions 9.0, 10.0, 11.0, 12.0 and 13.0.

Two things follow. It is a genuinely shared vocabulary — four competing search companies agreed on it, which is the only reason a term means the same thing to more than one consumer. And it is versioned and open, so it changes; a page written against it is written against a moving standard, not a fixed one.

The practical instruction is short: do not invent your own terms for concepts Schema.org already has. A bespoke property is invisible to everything except the code you wrote for it.

Third, put the claim in the markup, not only in the prose

This is the whole job, and most of it is unglamorous.

Say what the page is about. A page that never states its own subject in a machine-readable field is asking every consumer to infer it from the text, and inference is where they disagree with each other.

Make the important facts explicit. Anything a reader would want — what a thing is, who made it, when it was published, what it costs, whether it is available — belongs in a field, not only in a sentence a parser has to interpret.

Keep the structure and the prose in agreement. Markup that contradicts the visible page is worse than no markup: it makes the page unreliable rather than merely ambiguous, and unreliability is the one property that survives into every downstream system.

Fourth, check what a non-JavaScript client sees

Fetch your own page without executing scripts and read what comes back. This is a five-second check that reliably surprises people.

If the title, the description and the structured data are assembled in the browser, then the document that arrives at a client which does not run scripts contains none of them. Some clients render. Many do not, and the ones that do not fail silently — they get a valid, well-formed, empty-of-meaning document and proceed.

The same check answers the question people usually ask second: whether the markup is correct. A block of structured data that never reaches the response cannot be correct or incorrect. It is absent.

What survives to a model, and what does not

Search engine optimization, a field of study and professional skill, names backlinks and on-page SEO among its parts. Only one of those two is on your page. When a page’s text is read by a large language model rather than ranked by an algorithm, the external half does not travel: whatever the rest of the web says about you is not in the passage that was retrieved.

So the machine-readable head has a narrower and more literal job than the ranking-era version. It is not a signal that argues for the page’s importance. It is the page stating what it is, in a form that does not require anyone to be right about your HTML.

The order that works

Serve the important claims in the initial response. Use Schema.org terms rather than your own. Keep the markup and the visible page saying the same thing. Verify by fetching, not by believing. Everything else in this area is refinement on top of those four, and skipping any of them makes the refinements meaningless.

This site runs the same check on itself; that is the point of publishing it here rather than describing it in the abstract.

Retrieval-augmented generation and web indexing are both ways of finding an answer inside somebody else’s text. They are not the same technique, they do not fail the same way, and — this is the part that matters if you write pages — they do not reward the same page.

Two different things, precisely

Web indexing is a technology and a kind of algorithm, a subclass of indexing generally. It builds a structure over a collection so that a later lookup is cheap. The output of a lookup is a set of documents, ordered.

Retrieval-augmented generation is a subclass of information retrieval and of generative artificial intelligence at the same time. It is recorded as a facet of both generative artificial intelligence and the large language model, and it uses web search. That dual parentage is the definition doing real work: retrieval-augmented generation is not an alternative to retrieval. It is retrieval with a generator bolted to the end of it.

So the honest comparison is not “one or the other”. Indexing is a component; retrieval-augmented generation is an architecture that usually contains one. The interesting difference is what happens after the lookup.

What each one hands to a human

An index hands back documents. The ranking is the answer to “which of these is most likely relevant”, and a person does the rest — opens two or three, compares, decides.

Retrieval-augmented generation hands the retrieved passages to a large language model, an artificial intelligence model type and a kind of language model that uses prompts and the transformer architecture. The model composes a passage. The person reads one thing instead of ten.

That collapses a step the reader used to perform, and it moves a judgement from them to the system. Which passages made it into the context window is now a decision nobody sees.

What each one rewards on the page

Indexing rewards a page that is findable and distinguishable. It needs to be reachable, it needs terms that match what people ask, and it needs something that separates it from near-duplicates. A page can win here while being hard to read, because the human does the reading.

Retrieval-augmented generation rewards a page that is quotable in fragments. Retrieval usually operates on chunks, not whole documents, so the unit that competes is a passage. A passage that carries its own context — that names its subject rather than saying “it”, that states a claim rather than building to one over three paragraphs — survives being lifted out of its page. A passage that depends on the two paragraphs above it does not, because those may not have come along.

This is the concrete, unglamorous difference. Not a different keyword strategy. A different paragraph strategy.

Where they fail

An index fails visibly. You search, you get results, you can see that they are wrong and refine. The failure is in front of you.

Retrieval-augmented generation fails invisibly. If retrieval returns nothing useful, the generator still generates — that is what generators do. The answer arrives in the same confident shape as a good one, and nothing in it says “the retrieval step came back empty”. The person asking cannot distinguish a well-sourced answer from an unsourced one by looking at it.

For a page owner, that inverts the risk. Under indexing, being absent means being unfound. Under retrieval-augmented generation, being absent means something else gets said about your subject instead, and it might be wrong.

The older idea underneath both

The Semantic Web — an information system and a field of study, credited to Tim Berners-Lee and classified as a subclass of the World Wide Web, semantic network and semantic integration — proposed making the web’s meaning machine-available rather than only its layout. Its parts include linked data, which is also named as a quality of it.

Both techniques here are, in a sense, workarounds for that project not having been completed. Indexing infers meaning statistically from text and links. Retrieval-augmented generation infers it from text and a model. Neither has to guess about a fact that was stated explicitly and in structure, which is why explicit structure keeps paying under both regimes rather than one.

The practical summary

If you are choosing between them as an engineer, the answer is usually neither: you build retrieval and then decide whether to put a generator on it. If you are writing pages that both will read, the answer is to write passages that stand alone — name the subject, state the claim, keep the sentence unambiguous — because that page is the one an index can distinguish and a retriever can lift.

A web crawler is a software category. It is a subclass of Internet bot, and its documented uses are web scraping and web indexing. Everything else people say about crawlers — including most of what is currently said about AI crawlers — is built on top of that definition rather than contained in it, and it is worth separating the two.

What the definition actually contains

Three facts, and they are load-bearing.

A web crawler is a software category, not a single program. It names a class of things, which is why “the crawler” is usually the wrong noun.

It is a subclass of Internet bot. Its parent category is automated clients on a network, so anything true of Internet bots generally is true of it, and nothing about intelligence, intent or comprehension is implied by the classification.

Its uses are web scraping and web indexing. Web indexing is itself a technology and a subclass of both algorithm and indexing. So the crawler-to-index relationship is part of the definition: a crawler is a thing whose purpose includes feeding an index.

What a software agent is, separately

A software agent is a type of tool. It is a subclass of computer program, of tool, and of artificially intelligent entity, and it is said to be the same as a software bot.

Set those classifications beside the crawler’s and the difference is visible without inventing anything. A web crawler’s parent class is Internet bot. A software agent’s parent classes include artificially intelligent entity. Both are automated software; they are not classified under the same thing, and one of the two classifications mentions intelligence while the other does not.

That is a comparison of two definitions. It is not a claim that agents replace crawlers, that they behave differently on your server, or that one evolved from the other. Those may all be true. None of them is in the record, so this post does not assert them.

What a large language model is, separately again

A large language model is an artificial intelligence model type and a subclass of language model. It uses AI prompts and the transformer architecture. Its parts include the generative pre-trained transformer and generative artificial intelligence, its documented uses include conversational AI and computer security, and its opposite is the small language model.

Note what is absent: nothing in that definition connects a large language model to a web crawler. The phrase “AI crawler” joins two categories that the record does not join. It may name something real — but the joining is an assertion somebody is making, not a definition anybody can look up, and treating it as the latter is how a plausible sentence becomes a repeated fact.

Why this post is shorter than it wants to be

This is the honest part, and it is the reason the post exists in this form.

The obvious version of this article explains how AI crawlers differ from search crawlers, what that means for your server, and what you should change. Writing it requires relating a web crawler to a large language model, and no verified fact does that. Every sentence would have been an invention wearing the costume of a definition.

Search engine optimization illustrates the same restraint. It is a field of study and a professional skill, practiced by SEO specialists, sitting under internet marketing, digital marketing, marketing strategy and search engine marketing, and its parts include backlinks and on-page SEO. It is also recorded as different from generative engine optimization. All of that is about SEO. None of it says what a crawler does with a backlink, so this post does not either.

The summary

A web crawler is a software category and a kind of Internet bot, used for web scraping and web indexing. A software agent is a tool and an artificially intelligent entity. A large language model is an AI model type that uses prompts and transformers. Those are three definitions, and the interesting claims live in the space between them — which is exactly the space where nothing has been verified yet.

Generative engine optimization is a digital strategy for being found and used by systems that answer in prose rather than in links. It is a subclass of search engine optimization, and it is also listed as distinct from it — both of those are true at once, and the tension between them is the whole subject.

The definition, and the contradiction inside it

The knowledge graph records generative engine optimization as a kind of search engine optimization, and separately records the two as different from each other. That is not a bookkeeping error. It is what happens when a practice inherits its parent’s goal and replaces its parent’s mechanics.

The goal is unchanged: be the source a machine chooses when someone asks a question. The mechanics are not. Search engine optimization is a field of study and a professional skill, practiced by SEO specialists, and it sits under internet marketing, digital marketing, marketing strategy and search engine marketing. Its parts include backlinks and on-page SEO — signals that describe a document’s standing relative to other documents.

Generative engine optimization keeps the ambition and drops the ranking. Its main subjects are large language models and digital marketing. There is no results page to place tenth on. There is a passage that either survives into an answer or does not.

What a generative engine does with a page

A large language model is an artificial intelligence model type and a kind of language model. It uses prompts and the transformer architecture, and it has generative pre-trained transformers and generative artificial intelligence among its parts. Its stated uses include conversational AI. Its opposite, in the graph’s terms, is the small language model.

None of that describes a system that ranks. It describes a system that reads text and produces text. So the unit of success changes. A ranking engine asks which of these documents deserves position one. A generative engine asks which sentences it can safely repeat. Those two questions reward different pages.

A page optimized for the first question can afford to be thin, because its authority is carried by the links pointing at it. A page optimized for the second cannot borrow authority from anywhere. Whatever the model is going to say about you has to be present, in words, on the page it read.

Why structured data keeps appearing in this conversation

Structured data is a data type — a subclass of data whose defining quality is structure, and whose opposite is unstructured data. The graph is careful here: it is also different from semi-structured data, which is a real third category and not a synonym. Structured data uses a data model, and it is a facet of the Semantic Web.

That last relation is the one people skip. Structured data is not a ranking trick that happens to be machine-readable. It is a fragment of a much older project to make the web’s meaning available to software rather than only its layout. Generative engine optimization arrives at the same technique from the other direction, and for a blunter reason: a model that has to guess what your page means will sometimes guess wrong, and the wrong guess is what it tells the person asking.

What actually changes in the work

Three things, and they follow from the definitions rather than from anyone’s opinion.

Claims have to be findable in the text. A generative engine reproduces what it read. A fact that lives only in an image, a chart, a PDF behind a form, or the tacit knowledge of your sales team is a fact the model does not have and will either omit or invent.

Ambiguity is a cost you pay, not a nuance the reader enjoys. A human reader tolerates a sentence that could mean two things and picks the sensible one. A model picks one too, and it does not tell you which.

The competition is not the other ten results. There is no page two to be relegated to. There is inclusion in an answer and there is absence from it, and absence looks identical to never having published.

What it does not change

Generative engine optimization is a subclass, not a replacement, and treating it as a replacement is the most expensive mistake available here. The parent discipline still holds: a page nobody can reach, that loads slowly, that no other page references, is in trouble under either regime. Crawlability, clear information architecture and pages that genuinely answer something are prerequisites for both, not legacy concerns.

The honest summary is narrower than the marketing around it. Generative engine optimization is search engine optimization whose reader has changed from a system that ranks documents into a system that repeats sentences. Everything that follows — the emphasis on explicit claims, on structure, on saying the thing rather than implying it — is a consequence of that single substitution.