10/10/2026
Most of the work a business does to get recommended by AI sits in plain view. A clear website. Good descriptions. Consistent listings. Reviews.
Underneath all of it sits a small file most owners never open. It decides whether AI assistants' crawlers may read any of that work.
The file is robots.txt. It lives at yourdomain.com/robots.txt, and anyone can open it in a browser to see what it says.
Look for "Disallow: /" under GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Google-Extended. If it's there, those assistants are being told to stay away from your site.
Assistants favour businesses whose sites are easy for their crawlers to read. One blocking line works against everything else you've done to be described clearly.
Everything above the waterline only counts if the line below it lets assistants in. So before more effort goes into content, open the file and check what it says.