Show HN: Instantly generate /llms.txt to make your website AI-readable
Hacker News - AI
Jul 7, 2025 08:42
mandarwagh
1 views
hackernewsaidiscussion
Summary
A new web app has been launched to automatically generate /llms.txt files, a proposed standard similar to robots.txt but designed specifically for large language models (LLMs). This tool streamlines the process by scraping websites, extracting key Markdown links, and producing a ready-to-use llms.txt, making important content more accessible to AI systems. The growing adoption of /llms.txt by companies like LangChain and Mintlify highlights its potential to improve how LLMs access and understand web content.
Hey HN, I just built a small web app that helps you generate /llms.txt files for your website: https://llmstxt-cyuh.vercel.app GitHub: https://github.com/mandarwagh9/llmstxt What is /llms.txt? Think of it like robots.txt, but for large language models. It’s a new emerging standard that gives LLMs a clean, Markdown-based map of your most important content (docs, guides, APIs). Instead of crawling bloated HTML, they get straight to the point. The idea was popularized by Jeremy Howard and is gaining adoption across devtool companies like LangChain and Mintlify. Why I built this I wanted a dead-simple way to auto-generate /llms.txt from any docs site or URL. This tool scrapes your site, extracts key Markdown links, and gives you a copy-paste-ready llms.txt. No setup, no install, just paste your URL and download. Stack Frontend: Vercel + vanilla JS Backend: Uses Gemini API (but can swap in any LLM) Built with a focus on minimal UI + performance Would love feedback, feature ideas, or suggest