🚀 Secure control over AI crawlers - in minutes, flawlessly.
Generate LLMs.txt error-free now - free of charge
Advantages at a glance:
- Error-free creation thanks to integrated validation
- llms.txt and llms-full.txt in one go
- 100% free of charge, ready to use immediately
Why you need to act now: Loss of control of AI is imminent
LLM crawlers index the web - with or without your rules. Without clear instructions, you risk misinterpretation, outdated content in responses and loss of contextual authority over your brand. The fastest way to clean control is an error-free LLMs.txt and LLMs-full.txt - today.
Expert knowledge: Current support status of LLMs.txt (as of October 2025)
Transparent and realistic: LLMs.txt is not yet officially supported by the major providers. Individual experimental accesses are visible, but there is no formal confirmation. The concept is under discussion, but implementation is lagging behind. This status does not change the strategic need to provide clear, machine-readable guidelines - for future compatibility, governance and compliance. See sources at the end of this chapter in the section "Note: Current support status of llms.txt".
Fun fact
We have written and extended our version of the LLM.txt generator purely with AI!
If you want to learn how to properly deploy AI agents in large projects and keep AI rules on track, check out our 12 - week DEV AI Bootcamp!
The real pain: Why manual creation fails
Manual LLMs.txt creation sounds simple - but it is not:
- Prone to errors: formatting errors, inconsistent paths, forgotten pages.
- Time waster: 100 pages × 5 minutes = 8+ hours - and every update all over again.
- No validation: You only notice errors when it's too late.
Here are some impressions of the LLM SEO Text Generator
Simply enter the URL here
Which file do you want to generate?
Generation starts
When ready, simply download
Check preview
The second pain: Generic online tools disappoint
Many "free" generators only supply the bare essentials - often without them:
-
Complete crawling (sitemaps, JS rendering, depth control)
-
Consistency check between LLMs.txt and LLMs-full.txt
-
Integrated validation of the structure and links
-
Transparency on data protection and data sovereignty
-
And often: paywalls, limits, hidden costs
Result: Basic files without trust - and manual post-processing work again.
Resolution: obviousworks - the free online generator for error-free LLMs.txt
The logical answer to both pain points is a solution that combines consistency, full automation and validation - at no cost and with no setup.
-
100% free service
-
Fully automatic crawling of your domain (incl. optional depth and subpages)
-
Double consistency: LLMs.txt and LLMs-full.txt are generated synchronously
-
Integrated validation: structure, link and format checks
-
Data sovereignty: No permanent storage of your content
-
Instant download and clear deploy instructions
How online generation works in 3 steps
- Enter domain: https://www.obviousworks.ch/llm-text-generator open, enter domain.
- Select options: Include subpages, set depth, optionally activate AI descriptions.
- Generate & download: Download LLMs.txt and LLMs-full.txt, store in Webroot.
Tips for deployment:
- Provide files in the website root (e.g. https://deinedomain.ch/llms.txt)
- Set caching header moderately (max-age 1-7 days) and invalidate during updates
- For SPAs, ensure that the server delivers static files correctly
Competitive comparison: Self-hosting is possible - but secondary
Self-hosting can be useful if you need full process control or CI/CD integration. For most websites, however, the online generator is faster, easier and more secure.
| Tool | Perfect for | Not suitable if |
|---|---|---|
| obviousworks (online) | Immediate, error-free generation with validation | You absolutely need on-premise |
| obviousworks (self-hosted) | Regular updates in CI/CD, full control | You don't want a dev process at all |
| rdyplayerB | Maximum quality of descriptions is important | Budget for OpenAI API not available |
| Francesco | Quick start without external dependencies | AI enhancement desired |
| crawl4ai | Enterprise websites, custom requirements | You are looking for a simple solution |
| mendable | Integration into existing Python pipelines | You prefer a UI |
🚀 Generate online for free - without setup, in minutes
Optional: Self-hosting with obviousworks/ai-llmstxt-generator
If you want to run locally or in CI/CD, use the open source repo. Requirements: Node.js (≥18), npm/pnpm. Optional: OPENAI_API_KEY.
Clone # repository
git clone https://github.com/obviousworks/ai-llmstxt-generator
cd ai-llmstxt-generator
# Set variables
cp .env.example .env.local
# e.g.: OPENAI_API_KEY=sk-...
# dependencies
npm install
# start
npm run dev
Then open in the browser, enter domain, generate, export files. Details: README in the repo.
Note: Current support status of llms.txt (as of October 2025)
As of October 2025, the llms.txt file, which is similar to a robots.txt rules for access by AI systems, is not yet officially supported by the major LLM providers. The idea originally came from Jeremy Howard (Answer.AI), but its practical use remains theoretical. Search Engine Roundtable, PPC Land, WebYes
Current support status of the major providers
Current support status of the major providers
| Provider | llms.txt support | Source / Status |
|---|---|---|
| OpenAI (ChatGPT, GPTBot) | Not officially confirmed; individual server logs show experimental accesses | Search Engine Roundtable, PPC Land, WebYes |
| Google (Gemini, AI Overviews) | No support, according to Google employees such as John Mueller and Gary Illyes, llms.txt is ignored | Search Engine Roundtable, Stan Ventures |
| Anthropic (Claude) | No confirmed implementation, possible unofficial experiments | tryseo.de, WebYes |
| Microsoft (Copilot, Bing AI) | No known integration | tryseo.de, PPC Land |
| Meta (LLaMA, Meta AI) | No indication of support | PPC Land, WebYes |
| Perplexity AI | No official statement; currently no evidence in server logs | tryseo.de, WebYes |
Although individual blog posts or community pages claim that models such as ChatGPT, Claude or Gemini "respect llms.txt", this information cannot be substantiated by official statements or practical logging. WebYes, thellmstxt.com, IndexLab
Conclusion
- Fact: No major provider currently reads or pays attention to
llms.txt-files. - Experimental: Only a few OpenAI crawlers (e.g. GPTBot) appear to access these files on a test basis.
- Forecast: The concept is being discussed in the industry, but real implementations are still lacking.
Outlook: Why llms.txt could still prevail
- Parallels to
robots.txtSimple, decentralized mechanism with a history of success. - Regulatory pressure: data protection, copyright, governance.
- Ecosystem maturity: More tools (such as obviousworks) facilitate maintenance and adoption.
- Industry initiatives: Discourses and drafts drive standardization.
Conclusion: regain control - without effort
With obviousworks, you get maximum control, error-free files and real time savings - without costs, without setup, without risk. The most efficient way from 0 to "AI-ready" in minutes.
Fun fact
We have written and extended our version of the LLM.txt generator purely with AI!
If you want to learn how to properly deploy AI agents in large projects and keep AI rules on track, check out our 12 - week DEV AI Bootcamp!
Matthias (AI Ninja)
Matthias puts his heart, soul and mind into it. He will make you, your team and your company fit for the future with AI!
About Matthias Trainer profile
To his LinkedIn profile
FAQ
What is an LLMs.txt and why is it important?
The LLMs.txt is a machine-readable rules and metadata document that provides AI crawlers with information on how to use and interpret your website. It helps to structure content consistently, control access and ensure future AI compatibility and governance.
What is the difference between LLMs-full.txt and LLMs.txt?
LLMs.txt provides global rules and entry points. LLMs-full.txt goes deeper and contains detailed page descriptions (e.g. title, short description, categories). Together, they increase the consistency, context and usability of your content for AI systems.
Do major AI providers already officially support LLMs.txt?
As of October 2025, there is no official support from major providers. There are individual experimental accesses, but no broad standard yet. Nevertheless, preparation is worthwhile: clear guidelines, compliance and future compatibility.
What is the fastest and most error-free way to generate LLMs.txt?
Use the free online generator from obviousworks. It crawls your domain, validates the structure and links and generates LLMs.txt and LLMs-full.txt in minutes - without setup and with an integrated consistency check.
Is the online generator from obviousworks really free?
Yes, the service is 100 % free of charge. Optionally, AI descriptions can be generated with your own OpenAI API key. The generation also works without a key - only without AI descriptions.
What advantages does the obviousworks generator offer over generic tools?
Greater consistency, full automation, integrated validation, transparent steps, no hidden costs and instant download. Optional: AI descriptions and configuration options for depth/subpages.
How and where do I publish the files on my website?
Store LLMs.txt and LLMs-full.txt in the web root (e.g. https://deinedomain.tld/llms.txt). Set moderate cache headers (1-7 days) and invalidate during updates. Make sure that your server delivers static files correctly (also for SPAs).
Does the generator also work with JavaScript-heavy SPAs?
Yes, modern pages are supported. The generator takes dynamically rendered content into account and can optionally be supplemented with AI descriptions. Check the validation results after the crawl and adjust the depth/subpages if necessary.
Do I necessarily need an OpenAI API key?
No. The generator works without an API key. An OpenAI API key is only required if you want to generate optional, AI-supported page descriptions (AI descriptions).
How often should I update LLMs.txt and LLMs-full.txt?
Recommendation: For active websites weekly, for static websites monthly or after relevant changes. Use the online generator for quick re-runs and make sure to invalidate caches after updates.
What typical mistakes should I avoid?
Common errors: manual creation without validation, lack of consistency between LLMs.txt and LLMs-full.txt, outdated links, missing root provision and overly aggressive cache headers. The obviousworks generator prevents these errors with integrated checks.
Does LLMs.txt have an impact on SEO or rankings?
There is no evidence of direct SEO ranking effects. The primary benefit lies in governance, structuring and future AI compatibility. Indirectly, better machine readability can promote the use of your content by AI systems.
DE 
EN