AI Search

AI Crawlers Read Your HTML First, and Often Skip JavaScript

Vercel and MERJ measured 569 million GPTBot fetches against 4.5 billion Googlebot fetches in one month on Vercel's network.

On this page
  1. What the biggest crawler study found
  2. What a user-requested fetch sees
  3. A possible shift in September 2026
  4. How Google and Bing handle JavaScript and AI answers
  5. What OpenAI and Anthropic say about their crawlers
  6. What you can do about it
  7. Frequently asked questions

Can AI crawlers read JavaScript? Google renders JavaScript before indexing a page. But the best public tests found that most AI crawlers read only the HTML a server sends first. So text added by JavaScript may stay invisible to ChatGPT, Claude or Perplexity.

DJ Whiskers, the TygerLabs DJ tiger, at a laptop watching a small robot shine a flashlight on a mostly blank web page

What the biggest crawler study found

Vercel and MERJ analyzed AI crawler traffic on Vercel's network and nextjs.org in December 2024. The study reported that "none of the major AI crawlers currently render JavaScript," naming OpenAI's crawlers, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. ChatGPT's and Claude's crawlers do download JavaScript files, in 11.50% and 23.84% of their requests, but they do not run them. Gemini uses Google's own infrastructure and gets full JavaScript rendering, and AppleBot also renders JavaScript. For scale, in one month on Vercel's network, Googlebot made about 4.5 billion fetches. GPTBot made 569 million in that same period. Claude made 370 million, and PerplexityBot made 24.4 million.

What a user-requested fetch sees

In January 2026, the SEO agency RESONEO tested how "user-mode" fetchers behave when a person asks an assistant to open a page. ChatGPT-User and Claude-User saw static HTML, noscript content and template content, but they missed text that JavaScript inserted after the page loaded. Perplexity-User returned nothing at all, though RESONEO marked that fetcher "BROKEN" at test time. Bing Copilot, by contrast, did see inline-JavaScript content. This test did not cover OAI-SearchBot, GPTBot, ClaudeBot or PerplexityBot, so it only tells us about these specific user-requested tools.

A possible shift in September 2026

A developer's post on DEV Community, published September 30, 2026, shared server logs from a Next.js website. The logs suggested that OAI-SearchBot and GPTBot started rendering JavaScript from September 25, 2026. The author found "no announcement from OpenAI" about any change and said he cannot prove cause and effect from server logs alone. This is one website's data, so it points to a possible shift.

How Google and Bing handle JavaScript and AI answers

Google Search Central explains that "Google Search runs JavaScript with an evergreen version of Chromium." It works through three phases: crawl, render and index. A page can wait in a render queue for "a few seconds, but it can take longer." Google still recommends server-side rendering or pre-rendering because it is faster and because "not all bots can run JavaScript." Google also only finds links that are <a> elements with an href, so links that work only through scripts can be missed.

Microsoft Bing's webmaster guidelines state that Bing and Copilot share "the same core crawling, indexing, and ranking foundation as traditional search." Bing also says images or video "should not be the sole source of information" on a page. Google Search Central adds that AI Overviews and AI Mode have "no additional technical requirements." The only requirement is that a page is indexed and eligible to show with a snippet.

What OpenAI and Anthropic say about their crawlers

OpenAI's developer docs explain that OAI-SearchBot is used "to surface websites in search results in ChatGPT's search features." OpenAI recommends allowing it in robots.txt. The docs say nothing about whether this crawler runs JavaScript.

The Claude Help Center explains that Anthropic runs three bots: ClaudeBot for training, Claude-User for user-requested fetches and Claude-SearchBot for search. Anthropic says disabling Claude-User or Claude-SearchBot may reduce your website's visibility in Claude's search results. Like OpenAI's docs, this page says nothing about JavaScript.

What you can do about it

Test your own website first. Open a page, choose "View page source" and search for your services, prices, phone number and city. If that information is missing from the source, tools that skip JavaScript may not see it. Ask your developer about server-side rendering or static generation for your main content, titles and navigation. Google recommends this for the reasons above. Use working <a href> links in menus and service lists. Check that your robots.txt file and firewall rules allow OAI-SearchBot and Claude-SearchBot. This matters if you want your website to show up in those search results. Avoid relying on images or video alone to carry your key facts.

Our guide to WordPress or custom code covers how to choose a platform. Our guide to blocking AI crawlers with robots.txt covers how those rules work.

Frequently asked questions

Does ChatGPT run JavaScript when it reads a website?

Tests through December 2024 and January 2026 found that ChatGPT's crawlers and its page-fetching tool did not run JavaScript. A September 2026 server-log report suggests this may be changing, though OpenAI has not confirmed it.

Can Google read JavaScript websites?

Yes. Google Search Central states that Google renders JavaScript with an evergreen version of Chromium. A page can sit in a render queue before it gets indexed.

How do I check what AI crawlers see on my website?

Open the page and choose "View page source," then search for your services, prices, phone number and city. If that information only shows up after the page loads, tools that skip JavaScript may not see it.

Is server-side rendering better for AI crawlers?

Google recommends server-side rendering or pre-rendering because it is faster for users and crawlers, and because not all bots can run JavaScript.

Will blocking GPTBot remove my website from ChatGPT search results?

GPTBot and OAI-SearchBot are different crawlers. OpenAI uses OAI-SearchBot to surface websites in ChatGPT's search features, so blocking GPTBot alone would not affect that.

Most AI crawlers still read the HTML a server sends first, even as some signs point to that starting to change. Putting your key words directly in the HTML remains the safest way to stay visible to them.

If you want a developer to check how your website renders for AI crawlers, visit our web development services page.

Sources

  1. The rise of the AI crawler, Vercel blog (with MERJ)
  2. AI Search crawler test results, RESONEO (SEO agency)
  3. Traffic from ChatGPT jumped! Is it because its crawlers now run JavaScript?, DEV Community (hisashispace)
  4. Understand JavaScript SEO basics, Google Search Central
  5. Overview of OpenAI crawlers, OpenAI developer docs
  6. Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center
  7. Bing Webmaster Guidelines, Microsoft Bing
  8. AI features and your website, Google Search Central

More in SEO, AEO and GEO

Free Website Score

Check Your Website Free

Paste your address. In about a minute, see your score out of 100 and the top fixes for your homepage.

  • Speed
  • Phones
  • Google
  • Leads
  • Trust
  • Code
  • Security
  • AI answers

Share this post

SEO, GEO & AEO

SEO Services for Search and AI

SEO, GEO, and AEO for search results, answer boxes, and AI.

See SEO, GEO & AEO