I wanted AI agents to find the right Loke Development page, understand what we do, and use the public tools without guessing. That was the real reason for this work. The score was only a way to spot missing pieces.
I checked loke.dev from the outside, added the missing public routes, and then checked the live responses. This is the short version of what changed and how I tested it. You do not need every item for every site.
1. Decide what the site should expose
I started by writing down the jobs a caller should be able to do: find the company and services, read the important pages without a visual browser, search published content, and inspect one public URL.
That list became a few stable URLs: /llms.txt for the site map, /index.md for the homepage in Markdown, /openapi.json for the API, well-known files for tool discovery, and /sitemap.xml for public routes.
The normal HTML site stays in place. These extra routes are a second door for text-first clients. Keeping each one small also made the work easier to debug.
2. Give the homepage a plain text door
The homepage normally returns HTML. If a client sends Accept: text/markdown, the Worker returns a Markdown version. There is also a fixed /index.md URL for clients that do not want to use content negotiation.
curl -sS -H 'Accept: text/markdown' https://loke.dev/
curl -sS https://loke.dev/index.mdThe Markdown response includes the title, description, canonical URL, update date, service links, and public interfaces. It is not just a random copy of the HTML.
The response also sends Vary: Accept, Accept-Encoding and no-store for the negotiated homepage. Without Vary, a cache could give HTML to a Markdown client or the other way around. If you cannot control the cache key, use a separate Markdown URL instead.
3. Keep the public API small
The public API has two read-only jobs. One searches published posts and projects. The other checks metadata from one public HTTP or HTTPS URL. OpenAPI describes the operations, inputs, responses, errors, and limits.
curl -sS https://loke.dev/openapi.json
curl -sS 'https://loke.dev/api/v1/search?q=cloudflare'
curl -sS 'https://loke.dev/api/v1/inspect-url?url=https%3A%2F%2Floke.dev%2F'The search endpoint returns published content only. The inspection endpoint checks things like title, description, canonical, robots, Open Graph, and selected headers. Neither endpoint writes content or needs a user token.
4. Describe MCP with real tools
The MCP card at /.well-known/mcp.json points to the Streamable HTTP server at /api/mcp. It lists two read-only tools: search_published_content and inspect_public_url. Each tool has a description and an input schema.
curl -sS https://loke.dev/.well-known/mcp.json
curl -sS https://loke.dev/mcp.jsonI also added read-only annotations and linked the card from the homepage response headers. That gives clients more than one way to find the same server. A name or logo alone would not tell a client what it can safely do.
5. Add browser tools only when they help
The HTTP MCP server works for clients that can connect over the network. WebMCP is a separate browser path, based on the WebMCP specification. I registered the same two safe actions in the browser when the API is available.
if (typeof window === 'undefined') return
const modelContext =
navigator.modelContext ?? document.modelContext
if (typeof modelContext?.provideContext === 'function') {
await modelContext.provideContext({ tools })
}The code does nothing when the browser does not support the API. It also catches registration errors so an experimental feature cannot break the public page. Tool input is checked again inside the tool.
6. Add DNS-AID as a small experiment
DNS-AID is described in an Internet-Draft, not a finished RFC. It uses an SVCB record to publish a place where an agent can start an HTTPS lookup. The record format follows RFC 9460.
_index._agents.loke.dev. 300 IN SVCB 1 loke.dev. alpn=h2 port=443The record points the index name at loke.dev and says that HTTPS/2 on port 443 is available. I did not invent a second agent protocol. The public HTTPS site and MCP server are the real endpoints.
dig +dnssec _index._agents.loke.dev TYPE64
dig +dnssec loke.dev DSDNSSEC was enabled at Cloudflare and the parent-zone DS record was added at the registrar. A validating Google resolver returned the record with the authenticated data bit and its RRSIG. DNS-AID is still work in progress, so treat this as an experiment and verify it from outside your DNS dashboard.
7. Publish crawler and content rules
The site publishes a content preference header and the same policy in robots.txt. The current policy is Content Signals: search is allowed, while training and agent input are not granted by the publisher.
Content-Signal: ai-train=no, search=yes, ai-input=noUser-agent: *
Allow: /
Disallow: /api/
Disallow: /go/
Disallow: /health
Disallow: /search
Disallow: /studioThese files state a preference. They do not protect private data. Real privacy still needs authentication and server-side checks. The robots rules keep API, redirect, health, search, and Studio paths out of the normal crawl surface while leaving public pages available.
8. Do not add fake authentication
One scan reported a lower score for the combined API, authentication, MCP, and skill category. The tempting fix was to add OAuth files with made-up issuer and token URLs. I did not do that.
RFC 8414 describes metadata for a real authorization server. RFC 9728 describes a real protected resource. The public loke.dev API is anonymous and read-only, so there is no token service to describe.
{
"resource": "https://loke.dev",
"scopes_supported": [],
"authorization_servers": [],
"notes": "The public Loke.dev endpoints are read-only and currently do not require a bearer token."
}A missing document is not always a bug. If the site later adds a protected client area or write operations, that is when real OAuth discovery and scopes should be added.
9. Connect the pieces
I checked the routes as a small graph, not as unrelated files. The homepage points to the text version, company guide, API catalog, skill index, MCP card, and access notes. The catalogs point to the real endpoints and tools.
This gives a client recovery paths. It can start at the homepage, llms.txt, the MCP card, or the DNS entry. It should never need to guess a hidden route.
10. Test from the outside
I used local checks first: JSON and Markdown parsing, Sanity schema checks, typecheck, tests, and a production build. Then I checked the deployed site with HTTP requests. I looked at status, content type, Link and Vary headers, Markdown output, MCP cards, and DNSSEC responses.
curl -sS -D - https://loke.dev/llms.txt
curl -sS -D - -H 'Accept: text/markdown' https://loke.dev/
curl -sS https://loke.dev/openapi.json
curl -sS https://loke.dev/.well-known/mcp.json
dig +dnssec _index._agents.loke.dev TYPE64One check found an old internal note that said ai-input=yes while the live policy said ai-input=no. I fixed the note instead of changing the live response to match stale documentation. This is why a rescan is not enough. You need to read the actual response and compare it with the code.
The tools I used to make this work
This was not one magic tool or one scorecard. I used different tools for the app, content, edge delivery, DNS, and final checks. Each one answered a different question.
Cloudflare runs the live site
The production loke.dev site runs as an Astro app inside a Cloudflare Worker. The Worker handles the normal HTML pages, the Markdown homepage response, the public API, MCP routes, security headers, and cache behavior.
Cloudflare also manages the loke.dev DNS. I used it for DNSSEC, the DNS-AID record, edge delivery, and cache settings. The Cloudflare Workers docs explain the basic model: a request reaches the Worker, the fetch handler decides what to return, and the result is served from Cloudflare infrastructure.
Sanity holds the content
Sanity is the CMS. Posts, topics, authors, and images live there instead of being hardcoded into the Worker. The site reads the published content when it renders a page.
When published content changes, a signed Sanity webhook calls the revalidation route. That route checks the signature, finds related URLs, purges the matching Cloudflare cache entries, and warms important pages. Sanity documents this kind of document webhook as a way to react to Content Lake changes.
A note about Vercel
Vercel is not the production host for this repository. The current deployment config points to Cloudflare, so I have not described Vercel as part of the loke.dev runtime. Vercel is still useful for teams that want Git-based deployments and generated preview URLs. Its deployment docs explain that each deployment can get its own URL for review. If your site runs on Vercel, put the same machine-readable files in the app or public folder and test both the preview URL and the custom domain.
The scan sites are diagnostics
Is Agentic and IsItAgentReady are not hosting platforms. They request the public site and report what they can discover. I ran them after the platform work to find missing routes or metadata, then checked each important result with curl, source inspection, or DNS queries.
Run the check from a terminal
The official Is Agentic CLI is the easiest repeatable check. Run it from the repo or from a CI job. The first command prints a short report. The JSON form is better when you want to pass the result to a script or a coding agent.
npx is-agentic loke.dev
npx is-agentic loke.dev --json > is-agentic.jsonIf there is no completed report, the CLI starts a scan and waits for the result. If a report already exists, it reads that stored result. The JSON output gives you the score, the checks that need work, the evidence, and the report URL.
Turn a finding into a repair prompt
The report page has a Prompt to improve action on findings. I used it as a starting point for a small code change, then checked the source and ran the site checks myself. A generated prompt is useful context, but it is not proof that a fix is correct.
If you want this inside an agent workflow, save the JSON file, give the agent the failed finding and the files it relates to, and ask for a small patch with a test plan. The official agent skill can be added with this command:
npx skills add vercel-labs/is-agenticAfter the patch, run the project checks, inspect the public response, and run the scan again. This keeps the prompt tied to a real finding instead of turning the work into a blind checklist.
Check the public files yourself
The scanner only sees what is public. I also used plain HTTP and DNS commands to check the files and records directly. These commands are handy when a report says a route is missing:
curl -I https://loke.dev/AGENTS.md
curl -I https://loke.dev/llms.txt
curl -I https://loke.dev/.well-known/ai-catalog.json
curl -I https://loke.dev/.well-known/mcp.json
dig +dnssec _index._agents.loke.dev HTTPSI did not find a public official npx command for IsItAgentReady. I used its web report, then checked the reported routes with curl and the DNS record with dig. That extra step caught cases where a local file looked right but the deployed response was different.
The two scan sites are useful because they look at the public site from the outside. They use different checks, so I treated both as clues and then verified the actual routes, headers, JSON, and DNS myself.
Problems I hit
DNSSEC was the easiest place to get a false sense of progress. Turning it on at the DNS provider was not enough. The registrar also needed the DS record before a validating resolver could trust the chain.
Caching was another trap. HTML, Markdown, JSON, and DNS do not have the same freshness rules. I made the negotiated homepage private, gave public JSON a revalidation window, and put explicit update dates in the text files.
The last trap was adding files only because a scan mentioned them. A fake OAuth document or an unsafe browser tool would make the site less trustworthy, even if the score went up.
A short checklist you can reuse
1. Write down the real public jobs: find, read, search, inspect, or call.
2. Add one plain text entry point and one stable Markdown URL.
3. Publish a small site guide with real links and clear boundaries.
4. Describe public JSON with OpenAPI and keep it read-only until writes are needed.
5. Add MCP or browser tools only for useful actions, with exact names and input checks.
6. If you try DNS-AID, add DNSSEC and verify the record from a validating resolver.
7. Add access metadata only when the matching service really exists.
8. Run local checks, deploy, then test the public responses again.
What I would do next
The next work is maintenance, not more files. Keep the public guides and OpenAPI examples in sync with the real site. Add a small post-deploy check for status, content type, links, and JSON shape. Run the scans again after meaningful route or header changes.
If Loke Development later adds private client data, paid data, or write operations, the access model must change with it. Until then, a small public read-only surface is easier to explain and safer to keep stable.
The main lesson is pretty ordinary: good agent support is mostly good web engineering. Clear pages, stable URLs, plain text, typed JSON, narrow permissions, signed DNS, and honest metadata help people too.