This post, sponsored by JetOctopus, shares insights on optimizing websites for AI-driven search engines like ChatGPT and Perplexity, beyond traditional Google SEO. It addresses key questions about AI bot crawling, adapting technical SEO strategies, and understanding the evolving search landscape. By 2026, much of your site’s search impressions will come from AI agents researching for users. Unlike humans, these machines prioritize factors such as HTML loading cleanly within 200 milliseconds, product pages accessible within four clicks, and content that precisely answers specific, often novel, nine-word questions absent from any keyword research. This insight is backed by server log data from hundreds of enterprise sites since mid-2025. **What’s Happening on Your Site:** Query lengths are increasing unusually fast; for example, 10-word queries grew 161% year-over-year due to AI “fan-out”—where a single user query is broken into many sub-queries by AI. While impressions for longer queries surged, click-through rates dropped sharply to 2. 26%, as AI reads and synthesizes answers without sending users to your site. These “phantom impressions” represent real AI evaluation of your content, meaning filtering them out of reports risks missing critical signals. **The Three Types of AI Bots and Their Impact:** 1. **Training Bots:** Crawl broadly, ignoring click depth; they confirm AI has seen your content but don’t guarantee user visibility. 2. **AI Search Bots:** Explore pages but rarely beyond 2-3 clicks from the homepage and visit only monthly. 3. **AI User Bots:** Triggered by real user searches in AI tools, visit content relevant to user queries, and drive actual AI visibility. Heavy crawling by training or search bots doesn’t mean your content features in AI answers. Without segmenting log data by bot type, you only see part of the picture. **SEO Signals Respected by LLMs:** - **Robots. txt:** Crucial, as most AI crawlers (ChatGPT, Claude, Gemini) follow it; PerplexityBot respects it partially but its user bot does not. Review and update robots. txt for AI access. - **XML Sitemaps:** Widely used by AI crawlers for URL discovery; keep them accurate. Signals like canonical tags, noindex directives, and LLM. txt files do not affect AI bot visibility—they ignore meta-tags and such files. **JavaScript Rendering:** Most AI crawlers don’t render JavaScript, so if key content loads client-side, AI sees empty shells. Server-side rendering is essential for universal accessibility.

Google Gemini is an exception, using Googlebot’s rendering tech. **Improving AI Accessibility:** AI search bots rarely crawl deep beyond three clicks, so improve internal linking to ensure valuable deep pages are reachable within four clicks. Pages visited by training bots but ignored by user bots are high-priority fixes, while frequently visited pages signal content areas to expand. **Optimizing for Long Fan-Out Queries:** 95% of AI-driven queries have zero search volume and consist of AI-generated sub-queries. To uncover opportunities, filter Google Search Console data for queries over 7 words with impressions under 50 and zero clicks—the “Fan-Out Opportunity Matrix. ” Structuring content for list, comparison, review, and pros/cons formats captures these queries. Product review queries surged 16, 000% from mid to late 2025, highlighting the importance of detailed, structured content. **Technical Audit Steps:** 1. Identify AI user bot traffic in server logs by filtering specific user agents; tools like JetOctopus assist with segmentation. 2. Audit deep page accessibility by checking HTML payloads, avoiding JavaScript-dependent content, and ensuring quick load times and shallow click depth. 3. Review robots. txt carefully to avoid unintentionally blocking AI bots or exposing sensitive content. 4. Map phantom impressions via GSC API exports to understand AI queries related to your content and prioritize response strategies. 5. Continuously monitor changes in AI bot activity, Core Web Vitals, and SEO performance using tools like JetOctopus Alerts for unified reporting. **New KPI—Technical Accessibility:** SEO success in 2026 hinges on whether AI agents can crawl, reach, and extract facts from any page—be it your 50, 000th product page—in under 200 milliseconds. If not, traditional SEO efforts lose impact as AI dominates search. The foundational step is analyzing logs thoroughly to guide all other optimizations. In summary, AI-driven search requires rethinking SEO: focus on technical accessibility, deep internal linking, clean server-rendered content, and optimizing for complex AI queries. Regular log analysis and adjustments informed by real AI bot behavior are essential to maintaining visibility in the emerging machine-mediated search ecosystem. *Image Credit: JetOctopus (used with permission)*