--- title: 'A Markdown version of every page, for AI agents' url: 'https://www.dealmobile.co.uk/writing/a-markdown-version-of-every-page' type: Post published: '2026-10-23' summary: 'Why every page on this site now has a Markdown version for coding agents, the three ways an agent finds it, what goes into it, and how it stays out of search results and caches.' --- # A Markdown version of every page, for AI agents Published 2026-10-23 ![On the left, the top of a product page on this site, with the Markdown link in the masthead circled. On the right, the start of that page’s Markdown version: front matter with the title, address, type and summary, then the title as a heading and the first paragraph.](https://www.dealmobile.co.uk/sites/default/files/styles/almanac_shot_md/public/content/images/posts/a-markdown-version-of-every-page/hero.png.webp?itok=f1k_1iA1) More and more, someone who wants to try one of my products starts by pointing a coding agent at its page: read this and set it up for me. The agent fetches the HTML and has to dig the text out from among the navigation, the theme switch, the footer and the scripts. It mostly manages, but it spends its context on furniture, and it can miss the parts that matter, such as a screenshot’s alt text or the facts at the top of a product page. So every page here now has a Markdown version, made for agents to read. The Markdown link at the top of each page opens it, so you can see what an agent gets. ![The masthead of a product page: the name Tom Gould, the links About, CV, Products, Writing and Contact, a Markdown link with the Markdown logo, and the theme switch.](https://www.dealmobile.co.uk/sites/default/files/styles/almanac_shot/public/content/images/posts/a-markdown-version-of-every-page/masthead.png.webp?itok=-VdjKhf0) *The link at the top of every page. On the contact form, which has no Markdown version, it opens the index of every page instead.* ## Three ways in The first is the address. A page’s Markdown version is at its address with `.md` on the end, the convention from [llms.txt](https://llmstxt.org): this post’s is `/writing/a-markdown-version-of-every-page.md`. The front page has no path of its own, so its version is `/index.md`. The second is asking for it. Every HTTP request says which formats it would like in its `Accept` header. A browser asks for HTML first. Claude Code’s fetch tool asks for `text/markdown, text/html, */*`, which puts Markdown first, so at a page’s usual address it gets the Markdown version without knowing `.md` exists: ```bash $ curl -s -D - -o /dev/null -H 'Accept: text/markdown, text/html, */*' \ https://www.dealmobile.co.uk/products/turbopush | grep -iE '^(content-type|vary|content-location)' content-type: text/markdown; charset=utf-8 vary: Accept content-location: /products/turbopush.md ``` The third is links. Each page names its Markdown version in its head, as ``, and again in a `Link` header for tools that read only the headers. The link in the masthead and a line in the footer are for agents reading the page as text, and for people. ## An index of the site `/llms.txt` is the index. It has the site’s name and a line about it, then every page with a Markdown version: the writing, newest first, then the products, then the other pages, each as a link to its Markdown version with the page’s summary. An agent that starts there can find any page here with one request. The lists have Markdown versions too, and those aren’t split into pages. `/writing.md` has every post at once, where the HTML list shows ten at a time, and `/products.md` groups the products under the same headings as the products page. ## What goes in Each Markdown version starts with front matter: the title, the page’s address, what kind of page it is, the date for a post, and the summary. Then come the title as a heading, the main image with its alt text, and the content. A product’s facts come out as a short list near the top, because that’s the part of a product page an agent most often needs. This is the top of TurboPush’s, without the image: ```markdown # TurboPush Upload large files from the browser in chunks that survive dropped connections and page reloads, with a PHP endpoint that checks and reassembles them safely. - **Kind:** Library - **Status:** Maintained - **Started:** 2025 - **Licence:** MIT - **Stack:** TypeScript, PHP, Composer, tus, Jest, PHPUnit, Docker, GitHub Actions - **Links:** [Demo](https://turbopush.dealmobile.co.uk), [Source on GitHub](https://github.com/tomgould/TurboPush), [Package on Packagist](https://packagist.org/packages/tomgould/turbopush) ``` Each kind of page has its own list of what goes into its Markdown version, separate from what the HTML page shows. That’s how the furniture stays out: nothing is stripped from the page afterwards, because it was never put in. Links to other pages on the site point at their Markdown versions, so an agent that follows a link from a post to a product stays in Markdown. Images point at a smaller WebP copy. Every page ends with a link back to `/llms.txt`. The CV has a Markdown version too, built from the same data as the [JSON Resume copy](https://www.dealmobile.co.uk/cv.json). ## Keeping it out of the way The Markdown versions are for agents, not for search results. Each one is sent with `X-Robots-Tag: noindex` and a canonical link to its HTML page, so a search engine that finds one indexes the page instead. When one address can answer in two formats, any cache in between has to know, or a browser could be handed the Markdown an agent asked for a minute earlier. Both versions are sent with `Vary: Accept`, which tells a cache that the answer depends on that header. The site’s own page cache can’t mix them up: a request for Markdown is passed on as a request for the `.md` address, and each address is cached separately. ## What it misses Asking by `Accept` only helps agents that ask for Markdown. Plenty of tools send `*/*`, or put HTML first, and get the HTML page. For them, the links in the page are what point the way. A cache that ignores `Vary` would still mix the two versions up. Cloudflare, in front of this site, doesn’t cache its pages, so that isn’t a problem here. It would be if a cache rule started caching them. It’s read-only, too. An agent can read every page, but it can’t search the site or send the contact form, which is the one page without a Markdown version. ## Using it To point an agent at a product, give it the page’s address, the same one you’d open in a browser: ```text Read https://www.dealmobile.co.uk/products/turbopush and add it to this project. ``` Or start it at [llms.txt](https://www.dealmobile.co.uk/llms.txt), which links to everything else. From the command line, either of these gets a page’s Markdown: ```bash curl https://www.dealmobile.co.uk/products/turbopush.md curl -H 'Accept: text/markdown' https://www.dealmobile.co.uk/products/turbopush ``` - **Tags:** [AI](https://www.dealmobile.co.uk/writing/tag/ai.md), [Markdown](https://www.dealmobile.co.uk/writing/tag/markdown.md) --- Index of this site: [llms.txt](https://www.dealmobile.co.uk/llms.txt)