Verify Googlebot, Bingbot and GPTBot

To verify Googlebot, check the request’s IP address, not its user agent: either the IP is in Google’s published crawler list (common-crawlers.json), or a reverse DNS lookup gives a host under googlebot.com, google.com or googleusercontent.com that resolves back to the same IP. Bing and Apple document the same two methods. OpenAI publishes IP lists only, so GPTBot, OAI-SearchBot and ChatGPT-User are verified by their lists. This guide shows both methods with tested Python and Node.js code, plus an API call and a Cloudflare rule that do it for you.

Why verify crawlers

Many sites let search engines through rate limits, bot protection or paywalls. Scrapers know that and send Googlebot or GPTBot as their user agent. Any HTTP client can set any user agent, so the header alone proves nothing. Verification also keeps your analytics honest and stops you from blocking a real crawler by accident.

Official IP lists and DNS names

Every list below is a JSON file with a prefixes array of ipv4Prefix and ipv6Prefix entries in CIDR notation, the same format at every vendor. We checked each URL and documentation page on 2026-10-29.

Google’s older URL for the Googlebot list, developers.google.com/static/search/apis/ipranges/googlebot.json, now redirects to common-crawlers.json; update hard-coded URLs, and follow redirects when you download. The documentation pages are Google’s verifying Googlebot, Bing’s how to verify Bingbot, Apple’s Applebot page and OpenAI’s crawler overview.

To look at the current ranges, search them or download them as TXT, CSV, JSON or nginx config, use our pages for Googlebot, Google special-case crawlers, Bingbot, Applebot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and DuckDuckBot.

Method 1: check the official IP lists

Google recommends this method for large-scale checks: download the list, keep it in memory and match each request IP against it. The code reads the user agent to find which bot the request claims to be, then checks the IP against that bot’s list only:

# crawler_ips.py – Python 3.10 or later, standard library only
import ipaddress
import json
import time
import urllib.request

LISTS = {
    "googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
    "bingbot": "https://www.bing.com/toolbox/bingbot.json",
    "applebot": "https://search.developer.apple.com/applebot.json",
    "gptbot": "https://openai.com/gptbot.json",
    "oai-searchbot": "https://openai.com/searchbot.json",
    "chatgpt-user": "https://openai.com/chatgpt-user.json",
    "perplexitybot": "https://www.perplexity.ai/perplexitybot.json",
}
REFRESH_SECONDS = 24 * 3600
_cache: dict[str, tuple[float, list]] = {}


def _load(url: str) -> list:
    with urllib.request.urlopen(url, timeout=10) as res:
        prefixes = json.load(res)["prefixes"]
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes]


def _networks(bot: str) -> list:
    entry = _cache.get(bot)
    if entry is None or time.time() - entry[0] > REFRESH_SECONDS:
        try:
            entry = (time.time(), _load(LISTS[bot]))
            _cache[bot] = entry
        except OSError:
            if entry is None:
                raise  # no list yet; with an old one, keep using it
    return entry[1]


def claimed_bot(user_agent: str) -> str | None:
    """The bot a user agent claims to be, by the token each operator documents."""
    ua = user_agent.lower()
    for bot in ("googlebot", "bingbot", "applebot", "gptbot", "oai-searchbot", "chatgpt-user", "perplexitybot"):
        if bot in ua:
            return bot
    return None


def is_real_crawler(ip: str, user_agent: str) -> bool | None:
    """True: the IP is in the claimed bot's list. False: it claims a bot but isn't. None: no bot claimed."""
    bot = claimed_bot(user_agent)
    if bot is None:
        return None
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in _networks(bot))

We ran it with Python 3.15 against the live lists on 2026-10-29: 66.249.66.1 with a Googlebot user agent returned True, as did an IPv6 address from Google’s list and an address from the GPTBot list with the GPTBot user agent; a documentation address (203.0.113.9) with a Googlebot user agent returned False, and so did 66.249.66.1 claiming to be GPTBot. A browser user agent returned None.

The lists change without notice, so the code refreshes each one after a day and keeps the old copy if a download fails. The same check in Node.js, with the standard library’s BlockList:

// crawler-ips.mjs – Node.js 18 or later
import { BlockList, isIP } from "node:net";

const LISTS = {
  googlebot: "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
  bingbot: "https://www.bing.com/toolbox/bingbot.json",
  gptbot: "https://openai.com/gptbot.json",
  "oai-searchbot": "https://openai.com/searchbot.json",
  "chatgpt-user": "https://openai.com/chatgpt-user.json",
};
const DAY_MS = 24 * 60 * 60 * 1000;
const cache = new Map(); // bot -> { at, list }

async function load(url) {
  const res = await fetch(url, { signal: AbortSignal.timeout(10_000) });
  if (!res.ok) throw new Error(`${url}: HTTP ${res.status}`);
  const list = new BlockList();
  for (const p of (await res.json()).prefixes) {
    const [net, bits] = (p.ipv4Prefix ?? p.ipv6Prefix).split("/");
    list.addSubnet(net, Number(bits), p.ipv4Prefix ? "ipv4" : "ipv6");
  }
  return list;
}

export async function inCrawlerList(ip, bot) {
  let entry = cache.get(bot);
  if (!entry || Date.now() - entry.at > DAY_MS) {
    try {
      entry = { at: Date.now(), list: await load(LISTS[bot]) };
      cache.set(bot, entry);
    } catch (err) {
      if (!entry) throw err; // keep the old list when a refresh fails
    }
  }
  return entry.list.check(ip, isIP(ip) === 6 ? "ipv6" : "ipv4");
}

With Node.js 26 it gave the same answers for the same addresses.

Method 2: reverse DNS, then forward DNS

Google, Bing and Apple also document DNS verification. It needs no list, but two lookups per new address:

1. Reverse lookup of the IP: host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com.
2. Check the domain: googlebot.com, google.com or googleusercontent.com for Google, search.msn.com for Bing, applebot.apple.com for Apple.
3. Forward lookup of that name, and check that it resolves to the original IP. Without this step, anyone who controls the reverse DNS of their own addresses could name them something.googlebot.com.

# verify_dns.py – reverse DNS, then forward DNS back to the same address
import ipaddress
import socket

DOMAINS = {
    "googlebot": (".googlebot.com", ".google.com", ".googleusercontent.com"),
    "bingbot": (".search.msn.com",),
    "applebot": (".applebot.apple.com",),
}


def verify_by_dns(ip: str, bot: str) -> bool:
    try:
        host = socket.gethostbyaddr(ip)[0]  # reverse lookup (PTR)
    except OSError:
        return False
    if not host.lower().endswith(DOMAINS[bot]):
        return False
    try:
        forward = {info[4][0] for info in socket.getaddrinfo(host, None)}  # forward lookup
    except OSError:
        return False
    return ipaddress.ip_address(ip) in {ipaddress.ip_address(a.split("%")[0]) for a in forward}

Against live DNS, verify_by_dns("66.249.66.1", "googlebot") and verify_by_dns("157.55.39.1", "bingbot") returned True, and verify_by_dns("8.8.8.8", "googlebot") returned False (its name, dns.google, is not a crawler domain). DNS lookups block for up to a few seconds, so run them outside the request path or cache the result per IP; Bing’s guidance says a verified address can be cached for a long time. For OpenAI’s and Perplexity’s bots there are no documented DNS names: use Method 1.

Method 3: ask the IP API

If you already look up visitors with ipbase, pass the visitor’s user agent along. With the user_agent parameter, security.bot.spoofed says whether the address belongs to the operator of the bot the user agent claims to be: true for a fake, false for a real crawler, null when it cannot tell (no bot claimed, or a bot it doesn’t know). For bots whose operators document DNS names, such as Googlebot, bingbot and Applebot, the API also runs the DNS check for addresses outside the published ranges.

// spoofed.mjs – ask ipbase whether a request that claims to be a bot comes from that bot's operator
export async function botCheck(ip, userAgent) {
  const params = new URLSearchParams({ ip, user_agent: userAgent, fields: "security.is_known_good_bot,security.bot" });
  const res = await fetch(`https://api.ipbase.com/v2/info?${params}`, {
    headers: { apikey: process.env.IPBASE_KEY },
    signal: AbortSignal.timeout(3000),
  });
  if (!res.ok) throw new Error(`ipbase ${res.status}`);
  const bot = (await res.json()).data.security?.bot; // absent on plans without security data
  // spoofed: true = claims a bot it isn't, false = verified, null = cannot tell
  return bot?.spoofed ?? null;
}

The bot object also names the operator, the provider and the purpose (search, ai_training, ai_search, user_fetch and others) of a listed crawler. This is security data, part of the Medium plan and above; without it the function returns null for every request, which we checked against a local stub of the API. For many requests, send the user agent in the body of POST /v2/batch instead of the query string, so it doesn’t end up in URLs and logs. Details are in the info endpoint documentation.

Method 4: let Cloudflare do it

Behind Cloudflare, the rules language has a field for verified bots: cf.client.bot is true for requests from known good bots and crawlers on Cloudflare’s verified bot list. This custom rule (Security → Security rules → Create rule → Custom rules, action Block) stops requests that claim to be Googlebot but are not verified:

(lower(http.user_agent) contains "googlebot" and not cf.client.bot)

Cloudflare’s own bot-protection guides start with a rule that skips other custom rules for cf.client.bot, so verified crawlers are never caught by your blocking rules. Other CDNs have similar verified-bot settings.

AI crawlers and robots.txt

Verification tells you who is crawling; robots.txt tells the honest crawlers what they may do. The AI crawlers have separate user agents for separate purposes:

To stay in ChatGPT search and Google Search but opt out of model training:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

OpenAI says changes can take about 24 hours to reach its search systems. robots.txt is a request, not a lock: a scraper that fakes a crawler’s user agent ignores it, which is what the verification methods above are for. Before you block traffic by country, read the geo blocking guide: Google says Googlebot’s default addresses appear to be in the USA, so a site that blocks US visitors also blocks Googlebot’s US-based crawling.