Category
Utilities for working with AI output and AI crawlers
Small tools for jobs that come up around AI: fixing the JSON a model got wrong, and deciding which AI crawlers may use your website.
Start here
Pick a tool by what you need
Tools
Every tool in this category
- AI Crawler robots.txt GeneratorBlock or allow GPTBot, ClaudeBot and other AI crawlers.
- JSON RepairFix broken JSON from LLM output.
Coming next: Chat Format Converter, RAG Text Chunker.
About
How these tools fit together
Language models write JSON one token at a time, so their output can arrive wrapped in a code fence, with comments or trailing commas, or cut off at the output limit. JSON repair fixes what can be fixed, shows exactly what it changed, and warns when output was truncated, because a repaired cut-off answer is still missing data.
AI companies run several crawlers with different jobs: some collect training data, some build AI search indexes, and some fetch a page only when a user asks. Blocking all of them also removes you from AI answers. The robots.txt generator groups 27 crawlers by purpose, writes the file, and checks your current robots.txt with the same matching rules crawlers use (RFC 9309).
Both run entirely in your browser and show their working. JSON repair is built on the open-source jsonrepair library and lists every change as a diff, so you can check nothing was invented. The crawler list comes from each operator’s own documentation for its user-agent names and was last reviewed on 2026-10-08.
Read
Guides
FAQ
Questions people ask
Does robots.txt actually stop AI crawlers?
Only crawlers that choose to follow it. The major AI companies document that their crawlers respect robots.txt, but it is a request, not a lock. To enforce a block, use your host’s or CDN’s bot controls as well.
Will blocking GPTBot remove my site from ChatGPT?
Not by itself. GPTBot collects training data; OpenAI’s search crawler is OAI-SearchBot, and ChatGPT-User fetches pages when a user asks. The robots.txt generator lets you block training while allowing search.
Is it safe to paste my data into JSON repair?
The repair runs entirely in your browser; nothing is uploaded. Still, avoid pasting secrets into any web tool out of habit.
Why not just ask the model for valid JSON?
You should, using the provider’s structured output feature where it exists. Repair is the fallback for output that still arrives broken, for example when a response hits its length limit.