--- title: "Optimizing Your Website for AI Agents and LLMs" date: "2026-04-14" excerpt: "Your website has human visitors and AI visitors. Here's how to serve both — with llms.txt, inline LLM instructions, structured data, and machine-readable feeds." author: "Agnel Nieves" tags: ["AI", "SEO", "Web Development", "LLMs"] status: "published" lastModified: "2026-04-14" --- Your website has two audiences now. Humans, obviously. But also AI agents — LLMs that crawl, summarize, cite, and recommend your content to millions of people. If your site isn't optimized for both, you're leaving visibility on the table. I just finished optimizing [this site](/) for AI consumption, and the process revealed something interesting: most of what makes a site good for AI also makes it better for humans. Clear structure, machine-readable content, and explicit metadata benefit everyone. Here's what I did and why it matters. ## What Are AI Agents Actually Doing with Your Site? When someone asks ChatGPT, Claude, Perplexity, or Google's AI Overview a question, those systems don't just generate answers from training data. Increasingly, they fetch and cite live web content. Your site might get: - **Crawled for training data** by bots like GPTBot, ClaudeBot, and Google-Extended - **Fetched at query time** by Perplexity, ChatGPT browsing, and similar agents - **Cited as a source** in AI-generated responses - **Summarized in featured snippets** and AI overviews - **Navigated by autonomous agents** that interact with your APIs Each of these has different needs, but they all benefit from the same foundation: structured, discoverable, machine-readable content. ## The llms.txt Standard The [llms.txt spec](https://llmstxt.org) is the equivalent of `robots.txt` for AI agents. While `robots.txt` tells crawlers what they *can* access, `llms.txt` tells them what your site *is* — a structured markdown index served at your domain root. The format is simple: ```markdown # Your Name or Site > A one-line summary of what this site is. A longer description paragraph. ## Section Name - [Link Title](https://url): Description of what's at this link ``` I implemented two variants: - **`/llms.txt`** — the index. A table of contents with links to all pages, blog posts, projects, social profiles, and feeds. Think of it as a menu for AI agents to browse selectively. - **`/llms-full.txt`** — the full dump. Every blog post's complete markdown content, every project description, biographical context. For agents that want to load everything into context at once. Both are served as `text/plain` with markdown formatting. Both are generated dynamically from the same data sources that power the site, so they never go stale. ## Inline LLM Instructions in HTML This one comes from a [Vercel proposal](https://vercel.com/blog/a-proposal-for-inline-llm-instructions-in-html) and it's clever: embed AI-readable instructions directly in your page's `` using a script tag browsers ignore. ```html ``` Browsers skip `