Travel brands say they want AI to find them. Skift just checked fifty major travel sites and found most still will not tell an AI bot what to do. That is not caution. That is neglect dressed up as security.
On 9 November, Skift’s Adriana Lee published an audit of robots.txt files across fifty big travel websites. Only ten explicitly addressed AI crawlers. The other forty left AI to catch-all rules or defaults written for Google and Bing years ago. Brands are pouring money into generative engine optimisation while the file that actually greets the crawler still shrugs.
Visibility theatre versus the front door
The timing makes the gap sharper. On 30 October, Anthropic, Google, Microsoft and OpenAI backed a shared robots.txt opt-out for AI training. Catch-all rules do not carry it. If you want out of training, or in for search crawlers, you have to say so by name. Silence is not neutral any more. Silence is a decision you did not make.
Among the ten sites that did set AI rules, the pattern favoured visibility over training: Expedia and Vrbo open the door wide to AI search crawlers; Tripadvisor blocks training bots while allowing some search access; Best Western and ITC Hotels explicitly welcome OpenAI’s training crawler. Airbnb made the only major flip in the sample, moving from allowing GPTBot and ClaudeBot in August to blocking them across the board by October. That is at least a choice. Most of the industry has not even got that far.
Adobe’s July data already showed AI visits to travel sites up 119% year on year. Analyst Vivek Pandya’s warning was blunt: large chunks of US travel and retail sites remain unreadable to machines, so amenities, pricing and availability never reach the answer. Potential revenue sits on the table while marketers debate prompt packs.
The policy and the wall are not the same thing
Here is the part that should make CMOs sweat. Posted policy and live enforcement diverge. Eight sites in Skift’s sample, mainly hotels, returned 403 errors to automated requests even when their robots.txt looked open. CraftedStays found Marriott, Hilton, IHG, Hyatt and Four Seasons naming no AI crawler on their ban lists while bot protection still refused fetches. Cloudflare data put Meta’s crawler at 25.5% of AI-bot requests to travel and tourism sites — not OpenAI, not Anthropic.
That matches what Hospitality Net’s consolidated hotel-agent tests found in October: published files rarely name AI agents to block them, yet bot-management layers decide who reaches checkout. Marriott, IHG, Accor and Choice let agents through in both live tests. Hyatt and Hilton split by agent and route. Wyndham showed no rates. The guest’s AI cannot tell “sold out” from “blocked”. It just reports nothing and shops elsewhere — usually through an OTA that left the door open.
I have been saying the same thing about Tourism 5.0: the fight is not about prettier chatbots. It is about who the traveller’s AI can read, trust and book. If your content strategy for AI assumes the model can see your site, and your security stack quietly returns 403, you are funding a fantasy.
GEO starts with crawl permission, not clever copy
Too many travel marketers still treat GEO as a content workshop. Rewrite the FAQ. Add schema. Pitch “AI-ready” landing pages. Fine. None of that matters if GPTBot, ClaudeBot, PerplexityBot or OAI-SearchBot never get a clean 200 on the pages that hold your rates, policies and proof.
OTAs already behave like they understand this. Expedia’s own stack keeps opening agentic channels while hotels argue about scrapers. That is why Travel AI and GEO work has to start with crawl policy, bot allow-lists and agent-readable inventory — not another blog about “being cited in ChatGPT”. Citations without crawl access are theatre. Crawl access without a bookable path is free training for someone else’s cart. I made that point when Booking showed own-assistant conversion crushing LLM referral share. Visibility and checkout are different scoreboards.
So here is my take, bluntly: if you have a GEO budget and an unmarked robots.txt, you are paying for the wrong end of the funnel. Open a ticket with security and marketing in the same room this week. Decide which AI search crawlers you want in, which training crawlers you want out, and whether your WAF agrees with the file you publish. Then test it with more than one agent, from more than one network. Until the door and the policy match, every “AI visibility” slide is fiction.
Are you optimising for answers — or still hoping the bots guess correctly?
Want more takes like this on AI, search and travel marketing? Subscribe to my LinkedIn newsletter.