A firm can spend months getting its entity authority right, publishing detailed guides, and still discover that ChatGPT has never once mentioned it. The reason is rarely the content itself. It is often a single file, sitting quietly in the root folder of the website, telling every AI crawler that comes calling to turn around and leave.
That file is robots.txt, and it has quietly become one of the most consequential pieces of technical SEO in the AI search era. Most marketing teams have never opened it. Fewer still have checked whether it is blocking GPTBot, ClaudeBot or PerplexityBot by accident, alongside training crawlers they meant to keep out. Getting AI crawler access right does not guarantee a citation, but getting it wrong guarantees the opposite: your content simply cannot be considered.
The Technical SEO Check Almost Nobody Runs
Search engine optimisation has always rested on a basic assumption: if a page can be crawled, it can be found. AI search adds a second gate before that one. A large language model answering a question through retrieval only draws on sources its crawler was able to fetch in the first place, so a page can be superbly structured, rich in schema and written for exactly the query being asked, and still never surface in an answer because the crawler that would have fetched it was refused at the door.
How AI Crawlers Actually Reach Your Content
Not every AI-related crawler does the same job, which is exactly where confusion starts. Broadly, three types show up in a site’s server logs. Training crawlers gather content to build or refine a model and have no immediate bearing on what gets cited in an answer. Search or retrieval crawlers index content specifically to power live answers, which is the traffic that actually feeds citations. User-triggered fetchers retrieve a specific page only when someone asks an AI assistant about it directly, similar to a browser making a single request.
- GPTBot (OpenAI): gathers content for model training, with no direct role in citations.
- OAI-SearchBot (OpenAI): powers the citations that appear in ChatGPT’s search-backed answers.
- ClaudeBot (Anthropic): collects data used to train Claude’s underlying models.
- Claude-SearchBot (Anthropic): indexes pages specifically for Claude’s search-enabled responses.
- PerplexityBot (Perplexity): crawls and indexes content for Perplexity’s answer engine.
- Google-Extended: controls whether content can feed Gemini and AI Overviews, separately from standard Googlebot indexing.
Each of these crawlers can be allowed or disallowed independently in robots.txt. That granularity is useful, but it is also where most of the accidental blocking happens.
Why Blocking One Bot Often Blocks Another by Accident
Anthropic’s recent explanation of how Claude bots crawl sites clarifies exactly this problem: disallowing ClaudeBot stops your content from training future models, but it does nothing to remove you from Claude’s live, search-backed answers, because that traffic comes from the separate Claude-SearchBot and Claude-User agents. Anthropic published the clarification because so many site owners were applying one blanket rule and assuming it covered every Anthropic crawler. The same mix-up happens with OpenAI’s bots: a blanket rule against GPTBot, added purely to opt out of training data collection, will often catch OAI-SearchBot in the same disallow line if the robots.txt was written with a wildcard or copied from an outdated template, quietly switching off ChatGPT citations that the site owner never intended to lose.
What This Means for Singapore Businesses Right Now
The stakes are rising quickly for Singapore firms specifically. IMDA’s most recent Digital Economy Report found that 73.8% of Singapore’s workforce already uses AI tools as part of daily work, a figure that includes the people researching suppliers, comparing vendors and shortlisting agencies on a company’s behalf. When a procurement manager asks an AI assistant to shortlist AI SEO providers in Singapore, or a marketing lead asks Gemini to summarise structured data approaches, the businesses that appear are only ever the ones the crawler was allowed to read in the first place.
This is also where crawl access connects back to citation mechanics more broadly. As covered in how AI models choose citations, retrieval systems only ever draw from what they could fetch and index in the first place, so a blocked crawler removes a business from consideration before relevance, authority or structured data ever come into play. For a Singapore SME weighing where to spend a stretched marketing budget, an unintentional robots.txt block is one of the least defensible ways to lose ground, since it costs nothing to fix and nothing to notice once someone actually checks.
Auditing and Fixing Your Robots.txt
Fixing this does not require a developer sprint. A short, deliberate audit covers most of the ground:
- Pull the live robots.txt file directly from the domain root and read every disallow line literally, rather than assuming it does what a template comment claims.
- Check it against the current list of named AI crawlers published by each provider, since new agents are still being introduced.
- Decide policy per crawler type: many businesses are comfortable blocking pure training crawlers while deliberately allowing search and citation crawlers.
- Add an llms.txt file as a complementary signal about which content is most relevant, treating it as a supplement to standard crawl permissions and not a substitute.
- Re-check after any CDN, security plugin or firewall change, since several of these tools add their own AI-bot blocking rules by default.
A Deliberate Policy Beats a Default
None of this replaces the fundamentals of good SEO. Strong entity signals, clean structured data and genuinely useful content still decide whether a crawler that reaches a page finds anything worth citing once it arrives.
Crawl access is simply the precondition underneath all of it. Get it wrong, and none of the rest gets a chance to be judged, however good the content sitting behind that blocked line turns out to be.
Most robots.txt files were last touched years before ChatGPT existed, written for a web that only had to answer to Googlebot. Revisiting that file today is a small task with an outsized effect on whether AI platforms ever get the chance to recommend a business at all. If you would like a second pair of eyes on which bots your site currently allows in, or want a broader look at how your content shows up across AI search platforms, get in touch with the team at AISEOAgency.