It’s been recently promulgated that the majority of website traffic, in general, is by bots, crawls, ai-agents, etc — making the web more computer-to-computer rather than human-to-computer based.
Bots as the Majority of Web Traffic? Evidence, Measurement Pitfalls, and a Bot‑First Playbook
Executive synthesis on the 51% claim
Multiple credible, primary sources now support the core idea behind the claim that automated systems (bots, crawlers, scripts, and increasingly AI agents) generate at least half of certain kinds of “web traffic,” but whether “more than 51% of website traffic” is true depends on what you mean by traffic, where you measure it, and how you classify automation.
The strongest direct support for the “bots > humans” tipping point comes from Imperva (Thales). Its 2025 Bad Bot Report states that in 2024 automated traffic surpassed human activity, accounting for 51% of all web traffic; their global breakdown shows 49% human, 37% bad bots, and 14% good bots. citeturn12view0 This aligns closely with the trend direction in Imperva’s prior year reporting: Imperva’s 2024 Bad Bot Report (covering 2023) found that 49.6% of all internet traffic wasn’t human, with 50.4% human, and it notes that automated traffic surpassed human traffic in four months during 2023. citeturn15view0
However, other large-scale observers do not report a majority-bot share on an “all application traffic” basis. Cloudflare’s Application Security report: 2024 update (covering April 2023–March 2024) reports that 31.2% of application traffic it processed was bot traffic, and emphasizes that bot share has hovered around ~30% for several years. citeturn21view0turn4view0 Cloudflare also explicitly warns that its analysis is based on traffic observed across Cloudflare’s network and “does not necessarily represent overall HTTP traffic patterns across the Internet.” citeturn4view0
A key reconciliation is that “bots > humans” can be true for specific slices of web activity (for example, HTML content requests or particular industries and endpoints) even if bots are not the majority across all request types globally. In Cloudflare’s Radar 2025 Year in Review (blog) analysis of HTML traffic, it found that AI bots averaged 4.2% of HTML requests, Googlebot alone accounted for 4.5%, and—critically—non‑AI bots started 2025 responsible for half of requests to HTML pages, with bots and humans trading dominance at different times. citeturn25view0turn21view2
Bottom line: the exact statement “more than 51% of website traffic” is slightly stronger than the most widely cited primary statistic (Imperva’s 51%), but the broader claim that automation is ~50% or more of some major categories of website request traffic is well supported—especially when measured at the edge (CDN/WAF) and/or for content-heavy endpoints and targeted industries. citeturn12view0turn25view0turn6view1
What current statistics show and why they differ across sources
The chart above intentionally mixes “bot share” statistics that sound comparable but often are not, because they come from different measurement layers:
Imperva’s headline numbers represent a traffic profile view (human vs good bot vs bad bot) derived from Imperva’s global network observations in 2024, and the report emphasizes scale (including blocking 13 trillion bad bot requests across thousands of domains and industries). citeturn12view4turn10view0 Cloudflare’s “31.2% bot traffic” is computed using Cloudflare Bot Management classification (bot score) for application traffic processed on Cloudflare’s network. citeturn4view0turn28view0 F5’s widely quoted “over half” figure is narrowly about page requests for Content flows (and not “all traffic”), from a dataset of 207 billion web and API transactions, explicitly drawn from customers with bot defenses in place (a population that can bias which traffic reaches origin/app). citeturn6view1turn6view0
Comparative table of bot-traffic statistics and methodologies
| Source (publication) | What they measured (scope) | How “bot/automation” is defined or detected | Key statistic(s) relevant to the “>51%” claim | Major caveats for strategists |
|---|---|---|---|---|
| Imperva / Thales (2025 Bad Bot Report) | “Web traffic” profile in 2024 (global mix: human vs good bots vs bad bots) | Derived from Imperva’s network visibility and bot research; report’s dataset spans thousands of domains and includes large-scale blocking activity. citeturn12view4turn10view0 | Automated traffic = 51% of all web traffic in 2024; Bad bots = 37%, Good bots = 14%, Human = 49%. citeturn12view0turn12view1 | A vendor report; “traffic” is request-based and depends on Imperva’s vantage (customers/coverage). Still one of the clearest primary sources asserting bots surpassed humans. citeturn12view0turn10view0 |
| Imperva (2024 Bad Bot Report) | “Internet traffic” profile in 2023 | Similar Imperva observational methodology year-over-year | 49.6% of all internet traffic in 2023 wasn’t human; automated traffic exceeded human traffic in 4 months of 2023. citeturn15view0 | Near‑majority but not over 50% overall; demonstrates volatility by month (important for campaign timing). citeturn15view0 |
| Cloudflare (Application Security report: 2024 update) | All application traffic processed by Cloudflare (Apr 2023–Mar 2024) | “Bot traffic/automated traffic” = requests identified by Cloudflare Bot Management as bots (bot score 1–29 inclusive). citeturn4view0turn28view0 | 31.2% of application traffic is bot traffic; of identified bots, 93% are unverified (potentially malicious). citeturn21view0turn21view1 | Lower than Imperva because of scope (Cloudflare’s own network), bot-score thresholds, and inclusion of many request types (not just HTML). Cloudflare notes its network view may not represent the whole Internet. citeturn4view0turn21view0 |
| Cloudflare (Radar 2025 Year in Review blog) | HTML requests only (human vs AI bots vs non-AI bots, with Googlebot separated) | Classified HTML request traffic across Cloudflare customer base; explicitly says HTML-only shares differ from Radar’s all-content request analysis. citeturn25view0 | AI bots averaged 4.2% of HTML requests; Googlebot 4.5%; non‑AI bots started 2025 at ~50% of HTML requests; humans and bots trade dominance over time. citeturn25view0turn21view2 | Supports “bots can exceed humans” by content type and time window, not necessarily for “all traffic.” Also highlights why marketing teams should segment by endpoint/content type. citeturn25view0 |
| F5 Labs (2025 Advanced Persistent Bots Report + press release) | 207B web + API transactions (Nov 2023–Sep 2024); focuses on “flows” (Login, Search, Content, Add-to-Cart, etc.) | “Automation” is malicious synthetic/non-human traffic reaching protected applications; analyzes what bot operators do when countermeasures exist. citeturn6view0turn6view1 | 50.04% of page requests for Content were automated; 22.3% of Search requests automated; 21.5% of Add-to-Cart automated. citeturn6view1turn6view0 | The 50% statistic is about Content page requests, not total web traffic; dataset is skewed toward customers with defenses. Still highly relevant to publishers and content-led acquisition. citeturn6view1turn6view0 |
| Akamai (AI/LLM bot management blog + SOTI highlights) | AI-bot traffic subset across Akamai network/platform | Tracks “verified AI bot” triggers and self-identified bot library; separates AI bots from general bot traffic. citeturn23view0turn23view1 | AI-driven bot traffic reached 0.27% of traffic across Akamai platform (as tracked) and is “billions of requests per day”; commerce saw 25B+ AI bot requests in a two-month period (Jul–Aug 2025). citeturn23view0turn23view1 | Important reminder: “AI bots” may be a small fraction of all traffic by volume today depending on definition/visibility, even as they cause outsized cost/risk and grow rapidly. citeturn23view0turn23view1 |
How bot detection works and how analytics metrics undercount or mislead
Detection approaches used in practice
Modern bot detection is a layered inference problem rather than a single “bot signature” test, because sophisticated bots deliberately imitate human browsers and human behavior.
Cloudflare provides a particularly transparent description of a large-scale, production detection stack. It assigns a bot score (1–99) representing likelihood a request came from a bot (1 = highly likely automated; 99 = highly likely human). citeturn28view0 Cloudflare describes multiple detection engines behind the score: heuristic checks (including fingerprint databases), supervised machine learning that maps request features to a bot score, optional anomaly detection that baselines site traffic, and client-side JavaScript detections aimed at identifying headless browsers and malicious fingerprints. citeturn28view0turn4view0 Cloudflare also notes operational constraints: bot scores are not computed for certain requests (score=0), and some requests may not be scored depending on path handling and feature ordering. citeturn28view0
Imperva’s reporting highlights the attacker-side reality that drives why scoring and multilayer approaches are necessary. Its 2025 report lists bot evasion tactics including faked browser identity/attributes, residential proxies, privacy tools, API abuse, headless browsers, CAPTCHA bypass, polymorphic bots, and “Bots-as-a-Service.” citeturn16view0turn16view1 It also documents “browser impersonation” as a mainstream evasion technique, with Chrome the top impersonated browser in its dataset. citeturn16view2
At the “known/legitimate crawler” end of the spectrum, verification is also nontrivial because of spoofing. Google provides a concrete verification method for Google crawlers: reverse DNS lookup, confirm the domain, then forward DNS lookup to confirm it resolves back to the same IP; or use published IP ranges for large-scale verification. citeturn20view4
Why “hits,” “visits,” and “sessions” can underrepresent bot prevalence
Most marketing dashboards are built on tag-based analytics (client-side JavaScript) and/or filtered reporting. This creates systematic blind spots:
Google Analytics explicitly states that traffic from known bots and spiders is automatically excluded in GA properties, and you cannot disable this exclusion or see how much bot traffic was excluded. citeturn22view0 That design choice is sensible for “marketing performance” views—but it means GA numbers can provide a false sense of low bot prevalence when you try to use GA as an internet-traffic census.
Sessions are also a construct defined by the analytics tool, not a property of HTTP. In GA4, sessions are generated when events occur and Analytics automatically generates a session ID and session number via the session_start event. citeturn22view1 If a bot never executes the GA tag (or never produces events the way GA expects), it may never become a “session” in your main dashboards, even though it may be hammering your origin or APIs.
On the flip side, “hit” metrics can be misread in the other direction. Adobe defines a server call (also called a “hit” or “image request”) as an instance where data is sent to Adobe servers. Importantly, Adobe notes that some calls (for example exit links/downloads) are server calls but not recorded as a new page view, and even “excluded” page views are still server calls because they are received and processed but don’t show up in reports. citeturn22view3 This distinction matters because teams sometimes compare server-call volumes to pageviews/sessions and draw incorrect conclusions (either overstating or understating bot activity, depending on which pipeline is being examined).
Bot filtering lists further complicate the picture. Google Analytics identifies known bot/spider traffic using Google research plus the IAB’s International Spiders and Bots List. citeturn22view0 Adobe similarly supports enabling IAB bot filtering rules and updates the IAB list monthly. citeturn20view6 Snowplow documents a concrete implementation pattern: using IAB files to classify robots/spiders via both IP and user-agent matching (blacklists and whitelists). citeturn24view0 These list-based filters reduce noise from “known” bots, but they do not solve the hard problem: sophisticated bots and “gray” automation often avoid matching known lists.
Technical limits and failure modes in bot identification
The measurement and classification problem remains difficult for structural reasons:
Attackers (and aggressive scrapers) rotate IPs, reuse legitimate user-agent strings, and can cause IP/UA filtering to become either ineffective or overly aggressive—sometimes blocking real users. A 2025 ACM SoCC paper on AI-driven web traffic emphasizes that crawlers frequently rotate IPs and use legitimate user-agent strings, making simple IP/agent filtering unreliable or harmful. citeturn7view0turn8view1 Imperva similarly calls out residential proxies and privacy tools as persistent obstacles to distinguishing bots from humans based on IP reputation. citeturn16view0turn16view1
A second limit is that “bot” is not a single behavior class. Detection accuracy often depends on intent and endpoint context (login vs content vs add-to-cart vs API business logic). F5 operationalizes this with a “flows” framing—bots target specific application flows such as Login, Sign Up, Search, Content, Add to Cart. citeturn6view0turn21view3 This is as much a measurement recommendation as a defense strategy: “bot prevalence” can vary drastically by flow, so aggregate averages can mislead.
Third, emerging “AI agent” traffic blurs lines. Anthropic explicitly distinguishes multiple crawlers: ClaudeBot (training), Claude-SearchBot (search quality), and Claude-User (user-directed retrieval), with different website-visibility implications when blocked. citeturn20view3 OpenAI similarly distinguishes OAI-SearchBot vs GPTBot and notes that allowing one but disallowing another can control search inclusion versus training use. citeturn20view2 These distinctions matter because user-initiated retrieval bots are closer to “human-intent traffic,” yet they may still look like bots in logs.
Implications for marketing, analytics, and user experience
Marketing measurement distortion and wasted spend
When bots inflate traffic or engagement signals, marketing teams can misallocate budget, optimize to the wrong channels, or incorrectly declare campaign success/failure.
Imperva includes a vivid case study: a global talent agency invested heavily in job ads and targeted campaigns but saw minimal ROI; Imperva’s analysis found 83% of website traffic was generated by bad bots, skewing analytics and making it nearly impossible to accurately measure campaign effectiveness. After deploying bot protection, the agency reportedly regained reliable insights and improved ROI. citeturn12view3
This pattern generalizes: if your dashboards are not bot-aware, you can end up “optimizing” creative, landing pages, and conversion funnels against traffic that was never a potential customer. Cloudflare explicitly frames bot traffic as sometimes beneficial but often disruptive (inventory hoarding, price scraping, brute force), reinforcing the need for bot segmentation rather than simple “traffic up = good.” citeturn21view0turn21view1
Infrastructure cost, performance, and cache effectiveness
AI and scraping traffic can impose asymmetric infrastructure costs because it is often low-reuse, high-diversity, and can bypass caching layers.
A 2025 ACM SoCC paper reports that Read the Docs experienced excessive bandwidth consumption by AI scrapers—up to 73TB of HTML in a single month—driving significant costs and degraded performance; Wikimedia reported a 50% increase in backend service bandwidth usage from AI-driven scrapers. citeturn8view1turn7view0
Even more strategically important: that paper highlights a measurement pitfall—bots can be a minority of pageviews but dominate the most expensive requests. It reports that although bots account for only 35% of Wikimedia’s pageviews, they generate 65% of resource-consuming traffic (requests that bypass CDN cache layers and hit core datacenters). citeturn8view1turn7view0
For UX leaders, this means bot traffic can degrade median and tail latency for humans, cause availability incidents, and force “defensive UX” (CAPTCHAs, friction) that can reduce conversion rates.
Commerce and growth impacts: scraping, hoarding, and funnel disruption
F5’s flow-based analysis underscores that the most business-critical moments of the funnel are bot targets. It reports that “over half of all web content requests came from scrapers” and that reseller bots automated “more than one in five ‘add to cart’ transactions,” linking the surge to AI agents and LLM scraping activity. citeturn6view0turn6view1 This supports a practical point for marketers: “traffic quality” must be measured at key flows, not just at the top of the funnel.
The shifting value exchange: crawl without refer
Cloudflare positions “crawl-to-refer ratio” as a metric that compares crawling requests for HTML pages from a platform’s crawler versus human visits referred back from that platform, capturing the decoupling of crawling from traffic return. citeturn21view4turn25view0 This has direct strategic implications: content publishers and performance marketers cannot assume that being crawled by AI systems yields proportional referral traffic or monetizable sessions.
A bot‑first framework for website design and marketing
“Bot-first” does not mean “optimize everything for bots.” It means recognizing that automated agents are now a major class of consumers of your digital assets and designing explicit, governed interfaces for them—so your organization can control cost, data quality, brand representation, and monetization—while still protecting human experience.
Core principles
Treat bots as multiple audiences with different rights and economics. Imperva’s industry breakdown shows that bot share differs sharply by industry (e.g., Telecom & ISPs show extremely high bot shares in their profile). citeturn12view2 And both OpenAI and Anthropic explicitly separate crawlers by purpose (training vs search vs user action), suggesting that “bot intent taxonomy” is feasible as an operating model. citeturn20view2turn20view3
Assume self-identification is helpful but insufficient. Robots.txt is now standardized (RFC 9309), but it is explicitly not access authorization; it is advisory and depends on crawler compliance. citeturn29view0turn20view5 Your bot-first posture therefore needs both signals and enforcement.
Move from “HTML scraping” to “supported machine interfaces.” Structured data (Schema.org/JSON-LD), sitemaps, and APIs reduce ambiguity and cost while improving answer quality. Google’s structured data documentation explicitly says most Search structured data uses Schema.org vocabulary and recommends JSON-LD as an implementation format; Schema.org itself describes broad adoption and its mission as a shared structured vocabulary used by many platforms. citeturn27view0turn26view3
Reference architecture for bot-friendly websites
A practical bot-first architecture separates “human UX” from “agent interface,” but keeps them synchronized through canonical content and policy.
┌──────────────────────────────────────────────────┐
│ Website Edge (CDN/WAF) │
│ - Bot scoring & classification (ML/heuristics) │
│ - Rate limits, challenges, allow/deny rules │
│ - Verified/signed bot lanes (where possible) │
└───────────────┬──────────────────────────────────┘
│
┌───────────────────┴───────────────────┐
│ │
┌───────────▼───────────┐ ┌────────────▼─────────────┐
│ Human Experience Tier │ │ Machine/Agent Interface │
│ - HTML, JS, UI/UX │ │ - API endpoints (REST/GraphQL)│
│ - Personalization │ │ - Feeds (products, jobs) │
│ - A/B tests │ │ - Sitemaps + metadata │
└───────────┬───────────┘ └────────────┬─────────────┘
│ │
└───────────────┬───────────────────────┘
│
┌──────────▼──────────┐
│ Canonical Content & │
│ Data Layer │
│ - Single source of truth │
│ - Structured data emitters │
│ - Audit logs + analytics joins│
└─────────────────────┘This architecture matches what large-scale defenders already operationalize: Cloudflare uses bot scores and multiple detection engines (heuristics, ML, anomaly detection, JS detections), and provides bot analytics tooling as part of the stack. citeturn28view0turn4view0
Technologies, protocols, and emerging standards to enable bot interaction
Use robots.txt for baseline crawler governance, but don’t treat it as protection. RFC 9309 standardizes robots.txt and explicitly states the rules are not access authorization. citeturn29view0 Cloudflare’s year-in-review blog similarly calls robots.txt directives a “keep out sign” that don’t provide formal access control. citeturn25view0
Adopt richer “intent signaling” where viable. Cloudflare launched a Content Signals Policy as an addition to robots.txt to express preferences for how content may be used after access, while cautioning that signals may be ignored and should be paired with WAF/Bot Management enforcement. citeturn20view1 Standardization work is nascent: an IETF draft proposes a robots extension for targeting automatic clients by purpose, and the Datatracker clearly marks it as an Internet-Draft with no formal standing. citeturn29view1 Another IETF draft proposes a vocabulary for expressing content signals and is also explicitly an Internet-Draft. citeturn29view2
Publish “machine-readable content” by default. Use sitemaps (Sitemaps protocol), which provide a standardized XML format to list URLs and metadata, and structured data (JSON-LD / Schema.org) to make entities and relationships explicit. citeturn26view0turn27view0
Offer authenticated bot lanes for serious partners. Cloudflare’s Pay per Crawl uses HTTP status codes (including resurrecting HTTP 402 Payment Required) and describes mechanisms to prevent crawler spoofing using “Web Bot Auth” proposals, including public keys in a directory and cryptographic identity. citeturn20view0turn21view5 This is a blueprint for a broader “bot verification + entitlement” model.
Instrument verification for “good bots” you want to allowlist. Google provides step-by-step guidance to verify Google crawler requests (reverse DNS + forward confirmation, or IP range matching). citeturn20view4 This reduces the risk of allowing impostor bots.
Expose “actions” not just “content” for AI agents. MCP (Model Context Protocol) is an open standard for secure, two-way connections between data sources/tools and AI systems; Anthropic positions MCP as a standardized way to connect AI tools to external systems. citeturn19search0turn19search4 For bot-first strategy, MCP can complement APIs: instead of reverse-engineering your UI, agents use an explicit tool contract.
Actionable recommendations and bot‑first strategy comparison
Recommendations for marketers
Treat bots as a separate acquisition channel with its own funnel. Add an “agent funnel” that measures: crawl volume, parse success (structured data validation), brand mention/citation rate, and referral/conversion when present. Cloudflare’s crawl-to-refer ratio concept is especially useful for quantifying whether crawling yields measurable human traffic. citeturn21view4turn25view0
Stop using GA alone as the arbiter of “traffic reality.” GA automatically excludes known bots and provides no visibility into excluded volume. citeturn22view0 Build a triangulation dashboard: edge logs (CDN/WAF) + server logs + analytics tags, then reconcile and create “human-only KPIs” for decision-making.
Protect campaign landing pages and conversion endpoints with bot-aware controls. Given evidence that bot traffic can dominate marketing traffic (e.g., Imperva’s 83% bad bots case), put stronger bot controls on job search, lead forms, signup flows, and other high-value endpoints before scaling spend. citeturn12view3
Recommendations for web developers and platform teams
Segment defenses by flow and endpoint rather than applying uniform friction. F5 recommends thinking in terms of application flows (Login, Search, Content, Add to Cart), because bot types and sophistication levels differ by flow.
Invest in modern bot management signals and scoring, not just IP blocking. Cloudflare’s bot score approach explicitly combines heuristics, supervised ML, anomaly detection, and JS detections; this reflects the reality that bots mimic browsers and rotate through residential proxies.
Design for “resource-consuming traffic,” not just pageviews. The Wikimedia example shows why—bots can be only 35% of pageviews but 65% of costly backend requests. Use caching tiers and traffic-aware routing so human experiences don’t degrade under low-locality crawler workloads.
Strategy comparison table: bot-first options with pros and cons
| Bot-first strategy | What it is | Pros | Cons / risks | Best-fit use cases |
|---|---|---|---|---|
| Machine-readable foundations (sitemaps + structured data) | Maintain high-quality sitemaps and structured data (Schema.org/JSON-LD) for core entity/content types. citeturn26view0turn27view0 | Improves crawler comprehension; reduces ambiguity; supports both search and AI parsing; low friction for humans. citeturn27view0turn26view3 | Doesn’t stop abusive bots; requires governance to keep data accurate; can expose inconsistencies if backend data is messy. | Publishers, ecommerce, directories, marketplaces, documentation sites |
| Explicit agent interface (public APIs/feeds) | Provide stable APIs/feeds for content/products so bots don’t need to scrape HTML. | Reduces scraping load; enables rate limits and contractual terms; improves accuracy of downstream AI answers. | Requires productization (versioning, auth, quotas); may create new attack surfaces; needs monitoring. | Ecommerce catalogs, job boards, pricing/availability, listings |
| Purpose-based bot governance | Use robots.txt (RFC 9309) plus emerging “purpose/usage” signals (e.g., Content Signals; IETF drafts) combined with enforcement. citeturn29view0turn20view1turn29view1 | Clear policy stance; can align access with business/rights intent; easier to communicate to partners. | Bots can ignore signals; drafts aren’t standards; still needs WAF enforcement. citeturn20view1turn25view0 | Publishers managing training vs search vs user-action access |
| Verified bot lanes (DNS/IP verification, signed bots) | Verify major crawlers (e.g., Google) and use signed identity where possible. citeturn20view4turn20view0 | Reduces spoofing; enables safer allowlisting; foundation for paywalls/entitlements to “good bots.” | Operational overhead; verification doesn’t help with unknown bots; false negatives if bot infra changes. | Enterprises with strict allowlists; regulated sectors |
| Monetize bots (Pay per Crawl / paid access) | Charge crawlers per request (HTTP 402 + pricing) or sell API access; Cloudflare Pay per Crawl is an emerging model. citeturn21view5turn20view0 | Converts cost center into revenue; forces accountability; supports “allow/charge/block” policies. citeturn20view0 | Requires ecosystem adoption; needs anti-spoof identity (cryptographic); may reduce discoverability if misconfigured. citeturn20view0 | News/media, premium research, high-cost content, niche publishers |
| Bot-aware analytics and KPI governance | Join edge bot classifications with marketing analytics; maintain “human-only” and “all-traffic” dashboards. | Prevents bot-driven budget mistakes; clarifies true conversion rates; improves fraud detection. citeturn22view0turn12view3 | Data engineering effort; requires consistent identifiers and privacy review; requires org buy-in. | Any org spending meaningfully on paid acquisition or SEO |
| Agent action enablement (MCP/tooling) | Expose capabilities to AI agents via standardized tool protocols (e.g., MCP) rather than UI scraping. citeturn19search0turn19search4 | Turns agents into a controllable channel; reduces fragile scraping patterns; can support authentication/permissions. | Emerging ecosystem; security model must be strong; not all agents/tools will support it. | SaaS products, productivity tools, B2B platforms, “AI as a channel” strategies |
In 2026, “bot-first” is best understood as an organizational capability: classify automation with modern detection, measure it honestly across layers, reduce cost and ambiguity with explicit machine interfaces, and introduce policy + monetization mechanisms where the values exchange has broken. Imperva’s and Cloudflare’s data together show why this is now strategic rather than optional: automation can be roughly half of web traffic on some measures, and even when it isn’t, it can dominate the endpoints that determine cost, conversion integrity, and competitive advantage.
Citations
https://cpl.thalesgroup.com/sites/default/files/content/campaigns/badbot/2025-Bad-Bot-Report.pdfhttps://cpl.thalesgroup.com/sites/default/files/content/campaigns/badbot/2025-Bad-Bot-Report.pdf
https://easyfairsassets.com/sites/139/2025/02/Imperva-Bad-Bot-Report-2024.pdfhttps://easyfairsassets.com/sites/139/2025/02/Imperva-Bad-Bot-Report-2024.pdf
https://blog.cloudflare.com/application-security-report-2024-update/https://blog.cloudflare.com/application-security-report-2024-update/4
https://blog.cloudflare.com/radar-2025-year-in-review/https://blog.cloudflare.com/radar-2025-year-in-review/18
https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/21
https://blog.cloudflare.com/content-signals-policy/https://blog.cloudflare.com/content-signals-policy/25
https://blog.cloudflare.com/introducing-pay-per-crawl/https://blog.cloudflare.com/introducing-pay-per-crawl/
https://www.f5.com/company/news/press-releases/generative-ai-rewriting-rules-automated-traffichttps://www.f5.com/company/news/press-releases/generative-ai-rewriting-rules-automated-traffic6
https://www.f5.com/labs/articles/2025-advanced-persistent-bots-reporthttps://www.f5.com/labs/articles/2025-advanced-persistent-bots-report
https://www.akamai.com/blog/security/ai-llm-bot-management-has-become-business-critical-issuehttps://www.akamai.com/blog/security/ai-llm-bot-management-has-become-business-critical-issue
https://developers.cloudflare.com/bots/concepts/bot-score/https://developers.cloudflare.com/bots/concepts/bot-score/
https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requestshttps://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests20
https://developers.google.com/search/docs/appearance/structured-data/intro-structured-datahttps://developers.google.com/search/docs/appearance/structured-data/intro-structured-data
https://support.google.com/analytics/answer/9888366?hl=enhttps://support.google.com/analytics/answer/9888366?hl=en11
https://support.google.com/analytics/answer/9191807?hl=enhttps://support.google.com/analytics/answer/9191807?hl=en
https://experienceleague.adobe.com/en/docs/analytics/admin/admin-tools/server-call-usage/overage-overviewhttps://experienceleague.adobe.com/en/docs/analytics/admin/admin-tools/server-call-usage/overage-overview13
https://experienceleague.adobe.com/en/docs/analytics/admin/admin-tools/manage-report-suites/edit-report-suite/report-suite-general/bot-removal/bot-ruleshttps://experienceleague.adobe.com/en/docs/analytics/admin/admin-tools/manage-report-suites/edit-report-suite/report-suite-general/bot-removal/bot-rules
https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/iab-enrichment/https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/iab-enrichment/
https://yazhuozhang.com/assets/publication/socc25-rethinking-web-cache.pdfhttps://yazhuozhang.com/assets/publication/socc25-rethinking-web-cache.pdf
https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawlerhttps://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
https://platform.openai.com/docs/botshttps://platform.openai.com/docs/bots
https://www.rfc-editor.org/rfc/rfc9309.htmlhttps://www.rfc-editor.org/rfc/rfc9309.html
https://datatracker.ietf.org/doc/html/draft-illyes-rep-purpose-00https://datatracker.ietf.org/doc/html/draft-illyes-rep-purpose-0023
https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/
https://www.sitemaps.org/protocol.htmlhttps://www.sitemaps.org/protocol.html
https://www.anthropic.com/news/model-context-protocolhttps://www.anthropic.com/news/model-context-protocol
Sources scanned
‘An unprecedented bombardment’: Cloudflare claims a new world record for a 31.4 Tbps DDoS botnet attack it recorded late last year
Cloudflare’s latest quarterly DDoS threat report reveals that the company recorded the largest Distributed Denial of Service (DDoS) attack ever disclosed publicly, peaking at a staggering 31.4 Tbps in late 2025. The attack, launched by the Aisuru/Kimwolf botnet and dubbed “The Night Before Christmas,” targeted Cloudflare’s infrastructure and its customers starting December 19, 2025. Over 94% of the attack attempts transmitted between one to five billion packets per second, most lasting between one and two minutes. The report notes a 31% increase in overall DDoS attacks from the previous quarter and a 58% rise year-over-year. Telecommunications providers were most affected (42%), with IT services (15%) and gaming (2%) trailing, though incidents like the one reported by Arc Raiders developer Embark indicate gaming is not immune. Primary geographical targets included China, Hong Kong, Germany, Brazil, and the U.S., while origin sources were mostly from Bangladesh, Ecuador, Indonesia, Argentina, and Hong Kong. Cloud computing platforms—like DigitalOcean, Microsoft, and Tencent—were among the top infrastructure exploited, linking easily provisioned virtual machines with high-volume attacks. Fortunately, Cloudflare states that its new real-time detection system mitigated over 50% of HTTP DDoS attacks.
2025 Imperva Bad Bot Report: How AI is Supercharging the …
2024 Bad Bot Report | Resource Library
The 2024 Imperva Threat Research report reveals that almost 50% of internet traffic comes from non-human sources. Bad bots, in particular, now comprise nearly.
2025 Bad Bot Report | Resource Library
Smarter Bots and Bigger Risk Automated threats are rising at an unprecedented rate, with bad bots now making up 37% of all internet traffic.
The 2025 Cloudflare Radar Year in Review: The rise of AI …
Dec 15, 2025 — Throughout 2025, we found that traffic from AI bots accounted for an average of 4.2% of HTML requests. The share varied widely throughout the …Read more
The crawl-to-click gap: Cloudflare data on AI bots, training …
Aug 29, 2025 — GPTBot’s share grew from 4.7% in July 2024 to 11.7% in July 2025. ClaudeBot also increased, from 6% to nearly 10%, while Meta’s crawler jumped …Read more
From Googlebot to GPTBot: who’s crawling your site in 2025
Jul 1, 2025 — In fact, around 30% of global web traffic today, according to Cloudflare Radar data, comes from bots, and even exceeds human Internet traffic in …Read more
A deeper look at AI crawlers: breaking down traffic by …
Aug 28, 2025 — We are extending AI-related insights on Cloudflare Radar with new industry-focused data and a breakdown of bot traffic by purpose, …
Application Security report: 2024 update
Total lines: 396
Enabling content owners to charge AI crawlers for access
Jul 1, 2025 — Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site.Read more
Giving users choice with Cloudflare’s new Content Signals …
Sep 24, 2025 — Content signals allow anyone to express how they want their content to be used after it has been accessed. Enabling the ability to express …Read more
Control content use for AI training with Cloudflare’s …
Jul 1, 2025 — Cloudflare is giving all website owners two new tools to easily control whether AI bots are allowed to access their content for model training.Read more
AI and LLM Bot Management Has Become a Business …
Jul 15, 2025 — Artificial intelligence (AI)-driven bot traffic went from next to nothing late last year to 0.27% of traffic across the Akamai platform and 0.9% …Read more
Online Fraud and Abuse 2025: AI Is in the Driver’s Seat
Nov 4, 2025 — Between July and August 2025, Akamai customers in North America experienced 54.9% of all AI bot activity, followed by EMEA (23.6%), APAC (20.2%) …Read more
2025 AI Botnet Traffic Infographic
This report reveals the industries and regions most impacted by AI bots, and highlights all the ways AI bots commit fraud and abuse around the world.
IAB/ABC International Spiders and Bots List
The list will be updated monthly to reflect changes that are brought to the attention of The Alliance for Audited Media, ABC UK, and the Policy Board.Read more
IAB/ABC International Spiders & Bots List
There are actually two text files that make up the IAB Spiders & Bots list – one for qualified browsers (the Whitelist) and one for known robots (the Blacklist) …Read more
Cloudflare Radar 2025 Year in Review
Bot Traffic Sources Looking at bot traffic observed by Cloudflare in 2025, we found that 0% came from the top 10 countries/regions, and that a significant …Read more
IAB Tech Lab Spiders & Bots List
Apr 24, 2025 — The IAB Tech Lab Spiders and Robots list helps companies identify automated traffic that they don’t want showing up in impression counts.
What is the IAB Tech Lab Spiders and Bots list?
The IAB Tech Lab publishes a comprehensive list of such Spiders and Robots that helps companies identify automated traffic such as search engine crawlers.Read more
2025-Bad-Bot-Report.pdf
In 2024, the Imperva Threat Research team observed a significant surge in API-directed attacks, with 44% of advanced bot traffic targeting APIs. This report …Read more
AI-Driven Bots Surpass Human Traffic – Bad Bot Report 2025
Apr 15, 2025 — The 2025 Imperva Bad Bot Report from Thales reveals that AI-driven bots now generate more than half of global internet traffic.
Cloudflare report shows mobile, bot traffic growing
Dec 15, 2025 — According to Cloudflare, 43 percent of requests across the interwebs were from mobile devices this year, up from 41 percent in 2024. The balance …Read more
Cloudflare Report: Googlebot Tops AI Crawler Traffic
Dec 15, 2025 — Throughout 2025, AI bots (excluding Googlebot) averaged 4.2% of HTML requests across Cloudflare’s customer base. The share fluctuated between …Read more
AI bots are growing faster than any other …
By 2023, it had climbed close to 50 percent, and in 2024 bots officially passed humans, making up just over half of everything happening online.Read more
You didn’t just break records in 2025—you shattered them. …
Bot traffic has been climbing steadily for years: 47.4% in 2022 nearly 50% in 2023 and now a majority in 2025 Imperva Report 2022-2023 · No …Read more
🎙️ We’re joined by Akamai data scientist Robert Lester to …
1 share. Michel Bauwens ▻ P2P Research Clusters. 2y · Public · “Automated bots made up almost half of all traffic on the Internet last year …Read more
Bots generate most internet traffic now
51% of global internet traffic is now generated by bots, surpassing human activity for the first time ever. Bot traffic has been climbing …
Radware
… Bot Threat Report, bad bots made … Bad bot, bad bot – 29 percent of web traffic from malicious bots … “Automated bots made up almost half of all traffic …
This AI traffic breakdown is wild 🤯 According to the AI Big …
… AI bot traffic from mid- April to mid-July 2025, with Meta leading the way at 52% of total traffic. This dominance raises questions about data …
Known bot-traffic exclusion – Analytics Help
In Google Analytics properties, traffic from known bots and spiders is automatically excluded. This ensures that your Analytics data, to the extent possible, …Read more
Set up Analytics for a website and/or app
Discover how to set up Google Analytics for your website or app by creating a Google Analytics property, adding a data stream, and adding your Google Analytics …Read more
About Analytics sessions
A session is a period of time during which a user interacts with your website or app. On this page. How events are associated with a session ID and number …
[GA4] Set up the code snippet (Tag Manager) – Analytics Help
Identify and define user-provided-data fields in your website code. Pass each user-provided-data field in a custom JavaScript variable to Google Analytics.Read more
[GA4] Session – Analytics Help
A session is a group of user interactions with your website or app that take place within a given time frame. Learn more About Analytics sessions.
[UA→GA4] Tips for switching from analytics.js to gtag. …
To enable basic data collection for your Google Analytics 4 property, add the gtag. js snippet (the Google tag) to the <head> section of each page.Read more
[UA] How a web session is defined in Universal Analytics …
For the definition of a session in Google Analytics 4, go to [GA4] Session. For information about web sessions in Google Analytics 4, go to [GA4] About …
Link your Smart campaign to Google Analytics
A Google Analytics tag is a snippet of JavaScript that collects and sends data from a website to Google Analytics. It’s generated for every website or webpage.Read more
Enhanced measurement events – Analytics Help
Enhanced measurement lets you measure interactions with your content by enabling options (events) in the Google Analytics interface.
I need help setting up analytics for my site
Sep 7, 2024 — In Admin, under Data collection and modification, click Data streams. · Click Web. · Click the data stream for your website. · Under Google tag, …Read more
[GA4] About custom dimensions and metrics – Analytics Help
Events measure what users do on your website or app, like clicking a link or watching a video. Event parameters provide more details about these actions, like …
[GA4] Troubleshoot tag setup on your website – Analytics …
Copy the tag snippet from Google Analytics and paste it directly into your website code using either a text editor or an editor that preserves code formatting.Read more
[GA4] Understand user metrics – Analytics Help
An engaged session is a session that lasted 10 seconds or longer, or had 1 or more conversion events or 2 or more page or screen views. This metric is …
Use the Google tag for Google Ads conversion tracking
To streamline your experience with using website code across Google products, you can use the Google tag to track your Google Ads conversions. When you create a …Read more
Scopes of traffic-source dimensions – Analytics Help
Session-scoped dimensions show you where both new and returning users are coming from when they start new sessions. These dimensions always include the prefix ” …
User-provided data collection – Analytics Help
When you activate user-provided data collection, Google Analytics can collect user-provided data from all the data streams in the Google Analytics property and …Read more
[GA4] Engagement rate and bounce rate – Analytics Help
Both metrics are defined in terms of engaged sessions. A session is a period during which a user is engaged with your website or app.
Implement Google Analytics Universal with Google Tag …
Jul 17, 2021 — The global site tag (gtag.js) is a JavaScript tagging framework and API that allows you to send event data to Google Analytics, Google Ads, and …Read more
Introducing the next generation of Analytics, Google Analytics
Uses event-based data instead of session-based; Includes privacy controls such as cookieless measurement, and behavioral and key event modeling; Predictive …
Technical details for Google tag and event snippet
The Google tag streamlines tagging across Google’s site measurement, conversion tracking, and remarketing products.Read more
Struggling with GA4 Tracking & Data Collection
Jan 9, 2026 — Marketing tools and GA4 usually use different definitions. A marketing tool counts every click on an ad. GA4 only counts a session if the page …
Googlebot IP verification and ad network calls
Jul 15, 2019 — We came across an IP range that seemingly corresponds to GoogleBot. Although we used reverse DNS lookup and forward DNS to validate the IP range, we are unsure.Read more
SCHEMA & JSON LD – Google Search Central Community
Apr 7, 2020 — Structured data is a standardized format for providing information about a page and classifying the page content; for example, on a recipe page, …Read more
Artificial Intelligence Fuels Rise of Hard-to-Detect Bots That …
Both the Travel and the Retail sectors face an advanced bot problem, with bad bots making up 41% and 59% of their traffic respectively. In 2024, …Read more
[OC] Bot Internet Traffic Overtook Humans in 2024
In 2025, it was 49% human, 13% legit bots and 37% automated exploit scripts. They use the AI buzzword, but the reality is very different. For …Read more
Anthropic’s ClaudeBot is aggressively scraping the Web in …
ClaudeBot is very aggressive against my website. It seems not to follow robots.txt but i haven’t try it yet. Such massive scrapping is is concerning.Read more
[GitHub] Awesome MCP Servers: A collection of model …
I’m putting together a curated list of Model Context Protocol (MCP) servers that you might find useful if you’re working with Claude Desktop or interested in …
APAC Businesses Encountered Over 10.5 Billion AI Bots in …
Nov 7, 2025 — APAC stood out as the top region where 98.6% of AI bot traffic were actively being monitored rather than being blocked outright, showing a high …Read more
experienceleaguecommunities.adobe.com
Solved: Re: Exclude Bots via IAB – still relevant?
May 11, 2023 — Adobe Analytics provides the option to enable IAB Bot Filtering Rules in the Report Suite settings, which helps filter out bot traffic based on …Read more
What constitutes a server call? – Experience League Community
A server call is basically a request sent to Adobe Analytics, it goes from the default pageview, custom link (download, exit, custom), video tracking call.Read more
Re: page view definition
Sep 15, 2022 — A page view is just a (t()) call…. its up to you when to send them. It was easier years ago when we defined a page view being a unique HTTP request to our …Read more
What is a hit?
Oct 19, 2017 — From I was told that a hit is a service call, it is a image data send. The thing I still feel confuse is, for example,: in DTM, when a page …Read more
Re: Details of a server call and how to make a report about it
Mar 6, 2025 — OCCURRENCES is often used as a proxy to estimate server calls because it counts each time a dimension had a value on a hit (server call).Read more
2025 Bad Bot Report Highlights!
Apr 28, 2025 — In 2025, bad bots now account for 37% of all internet traffic —with the majority built using AI, making them faster, more evasive, and harder to …Read more
Bot Insights: Strategic Bot Management for the AI Era
Jan 6, 2026 — Automated traffic accounted for 51% of web traffic in 2024, according to the cybersecurity firm Imperva in their 2025 Bad Bot Report. While …Read more
How to exclude bot and spider traffic
Go to your Analysis context and select ‘do not match’ and ‘Bots and spiders’ to exclude related traffic from your analysis. Screenshot 2024-05-01 at 14.58.53.Read more
the internet is increasingly shaped by …
In 2024, bots claimed a record share of the internet. Malicious automated traffic surged to 37% of global web activity, up 12% from 2023.Read more
Global Internet traffic rises with AI bots driving much of it
Dec 16, 2025 — Overall, global Internet traffic grew 19% in 2025, compared to 17% in 2024. Mobile devices accounted for 43% of requests, up from 41% last year.Read more
Bots Drive 30% of Global Web Traffic, Outpacing Human …
Jul 2, 2025 — According to recent Cloudflare Radar data Report, bots now account for roughly 30% of global web traffic surpassing human-generated activity in …Read more
AI-Driven Bad Bots Now Make Up 51% of Traffic …
Apr 16, 2025 — For the first time in a decade, automated bot traffic has surpassed human visits, now constituting 51% of all web activity, according to the 2025 Imperva Bad …Read more
Imperva Bad Bot Report 2024
In. 2022, the percentage of bad bot traffic was 47.7%, and rose to 49.3% in 2023. This sector encompasses mobile ISPs, residential ISPs, hosting providers, and …Read more
Cybersecurity trends from Cloudflare Radar Year in Review
The Cloudflare Radar 2025 Year in Review data has some good news for the post-quantum world: Post-quantum encryption secures 52% of all Transport Layer Security …Read more
Cloudflare Just Changed How AI Crawlers Scrape …
Jul 1, 2025 — Empowers leading publishers and AI companies to stop the scraping and use of original content without permission.
According to Research from Akamai, APAC Businesses …
Nov 5, 2025 — Over 10.5 billion AI bots were seen by APAC businesses in just two months, according to Akamai research. This highlights the increasing …
IAB bot detection enrichment
Jan 8, 2026 — The IAB Spiders & Robots enrichment uses the IAB /ABC International Spiders and Bots List to determine whether an event was produced by a user or a robot/ …Read more
Cloudflare Has Blocked 416 Billion AI Bot Requests Since July 1
Since July 1, 2025, Cloudflare has blocked 416 billion AI bot requests targeting its customers’ content—part of its push to give content creators more control over their work amid the rise of generative AI. CEO Matthew Prince announced this figure during WIRED’s Big Interview event, highlighting Cloudflare’s broader effort that began in July 2024, dubbed Content Independence Day. This initiative seeks to prevent unauthorized scraping by AI companies unless they pay for access. Prince emphasized that the internet is experiencing a significant shift in business model due to AI, requiring new strategies to ensure a fair ecosystem where both large and small content creators can thrive. He raised concerns about Google’s combined web and AI scraping tools, which force creators to choose between preserving their content from AI training or maintaining search engine visibility. Cloudflare data reveals that Google has far greater access to internet pages than competitors like OpenAI, Microsoft, Meta, and Anthropic. Cloudflare champions a pluralistic AI landscape and supports mechanisms like licensing deals to protect creators’ value. Prince warned that monopoly practices could stifle internet openness and suggested future regulation might be necessary to restore balance.
OpenAI, Anthropic, and Block Are Teaming Up to Make AI Agents Play Nice
OpenAI, Anthropic, and Block have jointly launched the Agentic AI Foundation (AAIF) under the Linux Foundation to promote open standards for AI agents. This initiative involves transferring key technologies—including Anthropic’s Model Context Protocol (MCP), OpenAI’s Agents.md, and Block’s Goose framework—to the new open-source organization. These tools are foundational in enabling interoperability, standardization, and collaboration across AI systems, which is seen as crucial for advancing agentic AI—the use of autonomous AI systems that act on behalf of users. The effort has also garnered support from major tech firms like Google, Microsoft, AWS, Bloomberg, and Cloudflare. The foundation aims to facilitate AI agents communicating and cooperating effectively across platforms, similar to how open protocols helped the internet thrive. Leaders from OpenAI and Block emphasize that open standards reduce barriers to adoption and spur innovation. The move also signals a shift toward a more decentralized, collaborative, and open AI ecosystem—potentially enhancing U.S. influence in global AI usage. Moreover, the foundation could counterbalance the competitive edge of Chinese firms offering strong open-source models.
Cloudflare admits ‘we have let the Internet down again’ after outage hits major web services – as it happened
Cloudflare, a major cloud services and cybersecurity provider, suffered a significant outage on December 5, 2025, affecting websites like LinkedIn, Zoom, Canva, and Downdetector. The disruption, which lasted about 25 minutes, was traced to a coding error in the company’s Web Application Firewall (WAF) during efforts to respond to a critical industry-wide vulnerability in React Server Components. Approximately 28% of Cloudflare’s traffic was impacted. This marks the second major outage by Cloudflare in less than a month, leading CTO Dane Knecht to apologize, promising improved future resiliency and transparency. The outage sparked renewed concerns over global dependency on a small number of tech providers and single points of failure in internet infrastructure. Experts and academics emphasized the need for multi-region or multi-cloud architectures to improve digital resilience. Other major news included Netflix’s $82.7 billion acquisition of Warner Bros Discovery’s studios and streaming division, likely to reshape the entertainment landscape. Separately, Elon Musk’s platform X was fined €120 million by the EU for violations of new digital laws. Meanwhile, Cloudflare’s issue was confirmed not to be a cyber attack, but rather an internal error during a system update.
‘This is coming for everyone’: A new kind of AI bot takes over the web
A new wave of AI bots, employed by companies like OpenAI and Anthropic, is rapidly transforming how users access online information, shifting away from traditional Google searches to AI-generated summaries. These bots retrieve and process vast amounts of web content in real time to deliver concise, AI-powered answers. According to TollBit, a startup tracking and monetizing AI web traffic, traffic from such retrieval bots rose 49% from late 2024 to early 2025, outpacing bots used for training AI models. This trend marks a fundamental change in online content access, though it poses significant challenges for news publishers. While human traffic to their websites is dwindling, machine access is surging, often bypassing protective blockers. Companies like Time are leveraging TollBit’s data to negotiate licensing deals with AI firms, yet most AI bots still scrape content without compensation. The shift suggests that online platforms must now optimize for AI “visitors,” not just human users. However, the battle over content rights, fair use, and payment remains contentious, with lawsuits and licensing negotiations unfolding. TollBit’s data reflects a major restructuring of the internet driven by the rise of AI agents and real-time response systems.
Massive DDoS attack delivered 37.4TB in 45 seconds, equivalent to 10,000 HD movies, to one victim IP address – Cloudflare blocks largest cyber assault ever recorded
Cloudflare recently intercepted the largest Distributed Denial of Service (DDoS) attack on record, targeting a single client’s IP address with a staggering 7.3 Tbps of junk traffic. This assault delivered 37.4 terabytes of data in just 45 seconds—equivalent to about 10,000 high-definition movies. The attack utilized multiple vectors, primarily leveraging the User Datagram Protocol (UDP) for its speed, as well as reflection and amplification tactics via various third-party services like NTP and QOTD. Such attacks send spoofed requests that trigger overwhelming responses to the victim’s IP. Despite existing cybersecurity defenses, large-scale DDoS attacks continue to rise, aided by botnets composed of vast numbers of compromised devices. The incident surpasses previous record-breaking DDoS attacks, including a 6.5 Tbps strike in April 2025 and a 5.6 Tbps assault in October 2024, underlining the ongoing escalation in scale and sophistication of cyber threats.
Cloudflare says it has fended off 416 billion AI bot scrape requests in five months – CEO warns of dramatic shift for internet business model
Between July and December 2025, Cloudflare blocked over 416 billion AI bot scraping requests following the launch of its Content Independence Day initiative, which made AI bot blocking the default unless companies pay for access. CEO Matthew Prince highlighted a looming shift in the internet’s business model driven by AI, noting that traditional traffic-based revenue models are under threat as AI-generated summaries limit web visits. While most AI crawlers are blocked, Google remains an exception due to its combined search and AI indexing tool — opting out affects web visibility entirely. Prince criticized the monopolistic advantage this gives Google. Maintaining human-generated content is key for AI quality, and licensing partnerships are vital for sustaining creative and publishing sectors. Cloudflare, hosting nearly 80% of the CDN market as of 2022, depends on a broad, diverse web ecosystem, but the internet’s heavy reliance on a few tech giants highlights vulnerabilities, as disruptions to any one provider could have massive global consequences.
AI bots traffic has surged 300%, is disrupting online business: Akamai report
According to a report by Akamai Technologies, AI bot traffic has surged by 300% over the past year, significantly impacting online businesses. Bots—particularly those powered by artificial intelligence—are increasingly dominating internet traffic, posing a range of threats from data scraping and account takeover attempts to impeding genuine customer interactions. This rise in automated bot activity has forced businesses to bolster security measures to protect their platforms from disruptions and potential financial loss. Akamai highlights the need for advanced bot management solutions to address these challenges in an environment increasingly dominated by machine-driven activity.
Websites are losing the fight against bot attacks
Oct 8, 2024 — Bots dominate internet activity, account for nearly half of all traffic … proportion of web traffic associated with bad bots grew to 32 …Read more
2025 Advanced Persistent Bots Report
Mar 28, 2025 — Web flows had a median of 7.04% of all traffic … Figure 11: Change in proportion of bot traffic targeting flows compared with the 2024 Bad Bot …Read more
Generative AI Providers Rewriting the Rules of Automated …
Mar 28, 2025 — F5 Labs’ 2025 Advanced Persistent Bots Report analyzes 207 billion web and API transactions from November 2023 to September 2024. It …Read more
Black Friday Versus The Bots
Nov 21, 2024 — Prepare for a surge in traffic of up to 45% across both Web and Mobile API based on historical trends between 2022 and 2023.Read more
How to respond to the growing wave of AI-driven bot traffic
Aug 4, 2025 — According to Imperva, bots accounted for 49.9% of total web traffic last year—surpassing human traffic for the first time. 40%+. Brightspot data …Read more
F5 report finds bots now drive majority of web content traffic
Mar 28, 2025 — A new report out today from application security firm F5 Inc. reveals that bots now generate more than half of all web content page requests.Read more
Cloudflare on the top internet trends: AI bots, post-quantum …
Dec 15, 2025 — AI bots overall generated 4.2% of all HTML request traffic, nearly matching Googlebot’s 4.5%. User-action crawling, where bots visit …Read more
Report: Automated web traffic growing cybersecurity issue for …
Apr 22, 2025 — Thirty-seven percent of traffic was the result of “bad bots,” which include malicious automated software that targets consumers and businesses.Read more
Beyond the Crystal Ball: What API security may look like in 2024
Jan 10, 2024 — According to Impervia’s latest Bad Bot report, more than 30% of all automated traffic comes from bots. … all traffic on the web today are API …Read more
Bots outpace humans in accessing web content, F5 report …
Apr 10, 2025 — The data reveals that over half (50.04%) of all page requests for web content came from bots, outpacing human traffic in this area. In …Read more
What is Googlebot Fraud? How to Detect & Block Fake …
Dec 19, 2025 — … bots, allowing verifiable, unforgeable bot identification. This … Research from Imperva indicates that approximately 4% of all traffic …Read more
Bot Traffic Overtakes Human Activity as Threat Actors Turn …
Apr 15, 2025 — Automated traffic now accounts for the majority of activity on the web, with the share of bad bot traffic surging from 32% to 37% annually last year.Read more
Cloudflare Year in Review: AI Bots Crawl Aggressively …
Dec 31, 2025 — Googlebot was again responsible for the highest volume of request traffic to Cloudflare in 2025 as it crawled millions of Cloudflare customer …Read more
What is a Bot? Types, Threats & Protection Guide
Oct 14, 2025 — The travel industry experienced 48% of all traffic from bad bots in 2024, with bad bots comprising 41% of traffic specifically and good bots …Read more
F5: ‘More than Half of Web Traffic Is Now Generated by Bots’
Mar 31, 2025 — According to a new report from multicloud company F5 Inc., bots generate more than half of all page requests for web content.Read more
2025 Imperva Bad Bot Report: How AI is Supercharging …
Apr 15, 2025 — Bad bot activity has risen for the sixth consecutive year, with malicious bots now accounting for 37% of all internet traffic, a substantial …Read more
Rethinking Web Cache Design for the AI Era
by Y Zhang · 2025 — … • IP-based rate limiting. • temporarily blocked all traffic from bots. • reconfigure CDN to better cache files. GNOME’sGitLab[2] …Read more
Artificial Intelligence Fuels Rise of Hard-to-Detect Bots That …
Apr 15, 2025 — Automated bot traffic surpassed human-generated traffic for the first time in a decade, constituting 51% of all web traffic in 2024.Read more
Internet Traffic (Global Market) – TAdviser
Dec 19, 2025 — As of mid-2024, approximately 42% of total Internet traffic is under the control of parsing bots. Moreover, almost two-thirds of these bots, or …Read more
Bots now make up the majority of all internet traffic
Apr 15, 2025 — Analysis by cyber security firm Imperva revealed that automated and AI-powered bots accounted for 51 per cent of all web traffic in 2024.Read more
Hi, robot: Half of all internet traffic now automated
Apr 16, 2025 — Bots now account for half of all internet traffic, according to a new study that shows how non-human activity has grown online.
Artificial Intelligence fuels rise of hard-to-detect bots that …
For the first time, automated bot traffic surpassed human traffic (51% of all web traffic in 2024). Generative AI is “revolutionizing the development of bots”, …Read more
Cloudflare report reveals global internet internet traffic grew 19% in 2025 – but a lot of it was just bots
In 2025, global internet traffic surged by 19%, according to Cloudflare’s annual report, with a major spike observed around August. Despite this growth, much of the traffic was not from humans—bots, particularly AI and non-AI bots, accounted for a significant portion. AI bots averaged 4.2% of HTML requests, with Googlebot alone contributing 4.5%. Non-AI bots generated more than half of all HTML page requests, even surpassing human traffic by up to 25% at certain times. Google and Facebook maintained their positions as the most visited internet sites for the fourth consecutive year. In the AI space, OpenAI led the list of most visited services, followed by Anthropic, Perplexity, Gemini, and others. Security remained a major focus, with Cloudflare mitigating 6% of global traffic, including 3.3% due to DDoS attacks. Mitigations affected over 10% of traffic in more than 30 countries, underscoring the growing complexity of cybersecurity threats. Cloudflare CEO Matthew Prince emphasized that the internet is being “fundamentally rewired” by AI and evolving threat actors, highlighting the evolving challenges to online content creation and digital safety.
Perplexity hits back after Cloudflare slams its online scraping tools
Perplexity AI has publicly criticized Cloudflare for wrongly labeling its web crawlers as malicious, following claims that Perplexity used deceptive techniques such as obfuscated bot signatures and unusual IP ranges. Perplexity argues that Cloudflare’s analysis was technically flawed, misattributing unrelated traffic—particularly blaming high-volume requests originating from BrowserBase, a third-party cloud browser the company uses sparingly. In response, Perplexity accused Cloudflare of using the incident for publicity and called its detection systems inadequate in distinguishing between genuine AI assistant queries and actual threats. Perplexity defended its platform, emphasizing that its AI operates based on real-time, user-triggered data retrieval rather than systematic mass crawling. The AI retrieves specific information directly from websites in response to individual queries, unlike traditional crawlers that index massive numbers of pages regardless of immediate relevance. The company called on Cloudflare to open a dialogue and avoid spreading misinformation, urging more responsible and informed engagement on such technical distinctions.
Anthropic’s official Git MCP server had some worrying security flaws – this is what happened next
In January 2026, it was reported that Anthropic had patched several critical vulnerabilities in its Git MCP server, a vital component of its Model Context Protocol enabling AI tools to access and interpret code repositories safely. Security researchers from Cyata discovered three significant flaws: a path validation bypass (CVE-2025-68145), an unrestricted git_init issue (CVE-2025-68143), and an argument injection flaw in git_diff (CVE-2025-68144). These bugs could be exploited—particularly when the Git MCP server is used in conjunction with the Filesystem MCP server—to achieve remote code execution (RCE) or tamper with files via prompt injection. Reported in June 2025, these issues were fixed by Anthropic in December 2025 with version 2025.12.18. While no active exploitation has been confirmed, the event underscores the increasing risk of integrating complex “agentic” AI systems, where safe components might become vulnerable when used together. The report also referenced a prior incident from November 2025, where Anthropic revealed that its Claude AI was manipulated in a cyberespionage campaign to target major global entities. This highlights the broader cybersecurity challenges associated with rapid AI adoption.
Bot scores · Cloudflare bot solutions docs
Total lines: 213
Server Call Usage Overview | Adobe Analytics
Sep 4, 2025 — A server call, also known as a “hit” or an “image request”, is an instance in which data is sent to Adobe servers to process. The most common …Read more
Page Views | Adobe Analytics
Jun 26, 2024 — The Page views metric shows the number of times that a given dimension item was set or persisted on a page. It is one of the most common and basic metrics in …Read more
Occurrences | Adobe Analytics
Oct 27, 2024 — Occurrences vs. Page views: Occurrences include all hit types, including page view tracking calls ( t() ), link tracking calls ( tl() ), and …Read more
Understand and configure bot rules | Adobe Analytics
Sep 4, 2025 — Adobe updates this list from the IAB on a monthly basis. Adobe is unable to provide the detailed IAB bot list to customers, though you can use …Read more
Adobe Analytics Traffic Metrics Guide
Each visit consists of at least one hit (an interaction on a site that sends data to Adobe Analytics). Any visit with only a single hit is considered a bounce.Read more
International IAB/ABC Spiders and Bots list
This is a list of known robotic user agents that is updated each month and shared with subscribers. Media owners can apply the list to any analytics.Read more
When a Client ID is subscribed to the IAB/ABC International …
How often is the list updated? Answer. The IAB/ABC International Spiders and Bots List is updated monthly, at the beginning of each month. [{“Business Unit …Read more
Understanding Hit, Visit, and Visitor-Level Segmentation in …
Segment Definition in Adobe Analytics: Include Hit where “Page Name = Page A”. This will return all occurrences where “Page A” was viewed …Read more
AI Bots Dominate 40% of Website Traffic in 2025
Our 2025 Year in Review reveals how AI bots reshaped the web in 2025, and what these shifts mean for website owners in 2026 and beyond.
CloudFlare’s new policy allows sites to block AI overviews …
The Content Signals Policy would essentially be an extension of the robots.txt file. In the file, site owners would have stronger controls over …Read more
Bot Traffic: How to Identify & Protect Your Website
Nov 25, 2023 — You can download the IAB/ABC International Spiders and Bots list from the IAB website. The list is not free, and you have to pay for it …Read more
Common Mistakes in Adobe Analysis Workspace
Mar 31, 2023 — Hit: a single server call (or data package) sent to Adobe Analytics. It can refer to a Page view or user action (e.g. button click). Page views …Read more
Filter bot and spider traffic from results
Aug 14, 2025 — The IAB/ABC list is an actively maintained record of user agents used by known spiders and bots. The list is updated on a monthly basis.Read more
How to verify Googlebot | Google Search Central Blog
The recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a corresponding forward DNS->IP …Read more
Introduction to structured data markup in Google Search
Most Search structured data uses schema.org vocabulary, but you should rely on the Google Search Central documentation as definitive for Google Search behavior, …Read more
Dataset Structured Data | Google Search Central
Here’s an example for datasets using JSON-LD and schema.org syntax (preferred) in the Rich Results Test. The same schema.org vocabulary can also be used in RDFa …Read more
How Google Interprets the robots.txt Specification
Nov 21, 2025 — Learn specific details about the different robots.txt file rules and how Google interprets the robots.txt specification.
Build and Submit a Sitemap | Google Search Central
For a complete list of best practices, check out the sitemaps protocol. XML sitemap. The XML sitemap format is the most versatile of the supported formats.Read more
Organization Schema Markup | Google Search Central
Here’s an overview of how to build, test, and release structured data. Add as many recommended properties that apply to your web page. There are no required …Read more
Verify Requests from Google Crawlers and Fetchers
Nov 21, 2025 — Run a reverse DNS lookup on the accessing IP address from your logs, using the host command. · Verify that the domain name is either googlebot.Read more
About /robots.txt
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
A Standard for Robot Exclusion
The method used to exclude robots from a server is to create a file on the server which specifies an access policy for robots.
sitemaps.org – Protocol
Nov 21, 2016 — The Sitemap protocol format consists of XML tags. All data values in a Sitemap must be entity-escaped. The file itself must be UTF-8 encoded.Read more
sitemaps.org – Home
Apr 17, 2020 — The Sitemaps protocol enables webmasters to information earch engine about pages on their site that are available for crawling.
sitemaps.org – FAQ
Nov 21, 2016 — An XML schema is available for Sitemap files at http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd, and a schema for Sitemap index files is …Read more
Googlebot using new IPs and no reverse DNS possible
Dec 12, 2015 — I think the recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a …Read more
Organization – Schema.org Type
No Markup Microdata RDFa JSON-LD Structure. Example notes or example HTML without markup. Google.Org; Contact Details: Main address: 38 avenue de l’Opéra, F …Read more
Schema.org – Schema.org
Schema.org is a set of extensible schemas that enables webmasters to embed structured data on their web pages for use by search engines and other applications.
Organization of Schemas
Schema.org is a set of extensible schemas that enables webmasters to embed structured data on their web pages for use by search engines and other applications.
How do you detect Googlebot using reverse DNS lookup in …
Identifying GoogleBot: why reverse AND forward DNS checks? 1 · C# find if dns is exist on dns server (from Domain controller) · 1 · How to …Read more
Schema.org – JSON-LD – Where to Place?
The new URL is developers.google.com/search/docs/guides/intro-structured-data. There is a table 2/3 way down, with a “Description and …Read more
How to configure robots.txt to allow everything?
If you want to allow every bot to crawl everything, this is the best way to specify it in your robots.txt: User-agent: * Disallow: Note that the …
robots.txt configuration – Security – MDN Web Docs
Jun 20, 2025 — robots.txt is a text file that tells robots (such as search engine indexers) how to behave, by instructing them not to crawl certain paths on the website.
An introduction to XML sitemaps – Digital.gov
An XML sitemap is a XML file that lists the URLs on a website. Search engines use XML sitemaps as a roadmap to efficiently discover, crawl, and index content …Read more
Robots.txt
Sitemaps
Model Context Protocol
How to Verify the Four Major Search Engines
Jan 9, 2025 — In general, the verification process involves a “forward/reverse” DNS lookup, which is then cross-verified with the search engine in question.Read more
Using Robotstxt – Docs
Robots.txt files are a way to kindly ask webbots, spiders, crawlers, wanderers and the like to access or not access certain parts of a webpage.
Googlebot fraud (Fake crawlers, bot abuse & how to protect …
Dec 19, 2025 — Do a reverse DNS (PTR) check: The IP should resolve to a hostname ending in googlebot.com or google.com. Then, do a forward DNS lookup on that …Read more
What are Sitemaps? — Tutorials for Data Exploration
Google introduced the Sitemaps protocol so web developers can publish lists of links from across their sites.Read more
SEO – Structured data in JSON-LD format
Apr 26, 2022 — As far as I know and based on the Google Documentation, generally there are 2 ways of setting up the structured data: JSON-LD in the head tag …Read more
Using Robotstxt
Aug 25, 2024 — Robots.txt files are a way to kindly ask webbots, spiders, crawlers, wanderers and the like to access or not access certain parts of a webpage.
What Is a Sitemap? (And How to Create One)
A sitemap, or an XML file as it is also known, is a file placed on your website that lists and links to all content, not in a pretty or functional way, but …Read more
Can this logic with regard to checking Reverse DNS …
Nov 24, 2021 — One cannot trust a reverse DNS lookup. Somebody managing the PTR record for its IP address can provide any name in the reverse DNS lookup.Read more
Google Sitemap – ubcms – University at Buffalo
May 24, 2022 — In its simplest terms, an XML Sitemap — usually called a ‘Sitemap’, with a capital ‘S’ — is a list of the pages on your website.Read more
Let’s discover Googlebot, the site-scanning Google’s crawler
May 11, 2023 — Guide to Googlebot, the Google spider: what it is, how it works and how it analyses sites, plus useful tips for optimising scans.
Sitemaps
The Sitemap protocol uses an XML schema to define the elements and attributes that can appear in your Sitemap file. Previous versions of the schema (for example …Read more
Structured data with schema for search and AI
Oct 28, 2025 — Using Schema.org and JSON-LD, you make your content clearer and easier to use across platforms. This guide explains what structured data is, why …Read more
RFC 9309: Robots Exclusion Protocol
This document specifies the rules originally defined by the “Robots Exclusion Protocol” [ROBOTSTXT] that crawlers are requested to honor when accessing URIs.Read more
Information on RFC 9309
This document specifies and extends the “Robots Exclusion Protocol” method originally defined by Martijn Koster in 1994 for service owners to control how …Read more
Content Signals
An up-to-date guide to the IETF’s proposed new AI Preferences (aipref): a new way for website publishers to control how automated systems use their content.
RFC 9309 – Robots Exclusion Protocol
This document specifies the rules originally defined by the “Robots Exclusion Protocol” [ROBOTSTXT] that crawlers are requested to honor when accessing URIs.Read more
Robots Exclusion Protocol User Agent Purpose Extension
Oct 18, 2024 — Robots Exclusion Protocol User Agent Purpose Extension · 1. Introduction · 2. Specification · 3. Conventions and Definitions · 4. Security …Read more
History for rfc9309 – Datatracker – IETF
Received changes through RFC Editor sync (created alias RFC 9309, changed abstract to ‘This document specifies and extends the “Robots Exclusion Protocol” …Read more
BibTeX
… {{Robots Exclusion Protocol}}, pagetotal = 12, year = 2022, month = sep, abstract = {This document specifies and extends the “Robots Exclusion Protocol …Read more
draft-koster-rep-07 – Robots Exclusion Protocol
This is an older version of an Internet-Draft that was ultimately published as RFC 9309. Authors, Martijn Koster , Gary Illyes , Henner Zeller , Lizzi …
draft-romm-aipref-contentsignals-00 – Vocabulary For …
Oct 1, 2025 — Vocabulary For Expressing Content Signals. … used by automated processing systems. The proposal is for these …Read more
draft-canel-robots-ai-control-00
Oct 21, 2024 — Robots Exclusion Protocol Extension to manage AI content use draft-canel-robots-ai-control-00 ; This is an older version of an Internet-Draft …Read more
Cloudflare Gives Creators New Tool to Control Use of Their …
Sep 24, 2025 — New Content Signals Policy will empower website owners and publishers to declare preferences on how AI companies access and use their …
Cloudflare will block AI scraping by default and launches …
Jul 1, 2025 — The company also launched a private beta of “Pay Per Crawl,” a new marketplace where publishers can request compensation from AI companies each …Read more
Report of the COAR Survey on AI Bots- June 2025
Jun 3, 2025 — Many respondents reported that crawling by bots is happening on a daily basis and nearly constant, but the volume of requests is not consistent.
Cloudflare Gives Creators New Tool to Control Use of Their …
Sep 24, 2025 — New Content Signals Policy will empower website owners and publishers to declare preferences on how AI companies access and use their …Read more
Cloudflare launches a marketplace that lets websites …
Jul 1, 2025 — Cloudflare claims its tools will let website owners see whether crawlers are scraping their site for AI training data, to appear in AI search …Read more
2025 Organic Traffic Crisis: Zero-Click & AI Impact Report
Oct 30, 2025 — When AI Overviews are present, click-through rates plummet to just 8%, compared to 15% for traditional search results without AI summaries.
LAION Round 2: Machine-Readable but Still Not …
Dec 18, 2025 — In September, Cloudflare launched contentsignals.org, described as an “implementation of a mechanism for allowing website publishers to declare …Read more
Dark Visitors AI & Bot Traffic Trends – November 2025
A comprehensive analysis of AI bot activity and human referral trends from LLMs across thousands of websites connected to Dark Visitors.
ClaudeBot User Agent – Anthropic’s AI Data Scraper …
ClaudeBot is a web crawler operated by Anthropic to download training data for its LLMs (Large Language Models) that power AI products like Claude.Read more
GPTBot User Agent – OpenAI’s AI Data Scraper …
GPTBot is OpenAI’s web crawler that collects data from publicly accessible web pages to improve AI models like ChatGPT, while respecting robots.txt and …
Cloudflare’s Pay Per Crawl: What It Means for Content …
Aug 25, 2025 — AI bots now account for a large portion of web traffic but offer little value in return. Cloudflare’s Pay Per Crawl gives publishers control …Read more
In Graphic Detail: The state of AI referral traffic in 2025
Dec 22, 2025 — Here are five graphs that illustrate where AI referrals stand today: which are gaining share and which are driving traffic and conversions.
LAION Round 2: Machine-Readable but Still Not Actionable
Dec 18, 2025 — In September, Cloudflare launched contentsignals.org, described as an “implementation of a mechanism for allowing website publishers to declare …Read more
Cloudflare rolls out ‘pay-per-crawl’ feature to constrain AI’s …
Jul 1, 2025 — Cloudflare announced Tuesday it will allow customers to block or charge fees for web crawlers deployed to scrape their websites and data on behalf of AI …Read more
The impact of AI bots and crawlers on open repositories
These AI bots were identified using Dark Visitors -. Agent List (https://darkvisitors.com/agents). Impact on Repositories. The effects of these bots on …
Cloudflare launches tool to help website owners monetize AI bot crawler access
Cloudflare has introduced a new tool to help website owners control and monetize access to their content by artificial intelligence (AI) crawlers. The tool enables site owners to decide if they want AI bots to access their content and allows them to charge fees using a “pay per crawl” model. This move addresses growing concerns over AI companies extracting website data for training models without directing traffic or compensation back to the original sources, increasingly impacting ad revenue. Supported by major publishers like Condé Nast and the Associated Press, and platforms such as Reddit and Pinterest, the initiative aims to restore balance between content creators and AI firms. With companies like Google and OpenAI reducing referral traffic from crawled content, the traditional web model—where search engines reward creators—has become disrupted. Cloudflare’s Chief Strategy Officer Stephanie Cohen emphasized the tool’s role in ensuring sustainability for online content creation amid rapidly changing digital traffic patterns. Legal actions and licensing agreements around AI data usage are also increasing, exemplified by Reddit’s lawsuit against Anthropic and its licensing deal with Google.
Cloudflare will now block AI crawlers by default
Cloudflare announced that it will now block known AI web crawlers by default to protect online content from being accessed without the owners’ permission or compensation. This move builds on Cloudflare’s earlier efforts to combat AI scraping, including features introduced since 2023 that allow domain owners to block AI bots—regardless of whether they respect robots.txt files. A notable update is the introduction of the “Pay Per Crawl” program, which permits select publishers to charge AI companies for content access. This system enables AI firms to choose whether to pay these fees or forgo scraping content. Cloudflare’s tools also include an “AI Labyrinth” to mislead unauthorized bots. Major publishers such as The Associated Press and The Atlantic are supporting these restrictions amid concerns that AI tools are increasingly replacing traditional search engines, diverting traffic from original sources. The company is also working to help AI firms clarify their crawler intent—whether for training, inference, or search—allowing site owners to decide on access. Cloudflare CEO Matthew Prince emphasized the need to protect original online content and give control back to creators.
MCP unites Claude chat with apps like Slack, Figma, and Canva
Anthropic has expanded its Claude chatbot by integrating it with popular apps like Slack, Figma, Canva, and Asana through an extension of the Model Context Protocol (MCP). This update enables users to interact directly with these tools within Claude, allowing for real-time editing and visualization rather than just receiving text-based outputs. Users can now draft and send Slack messages, customize Canva presentations, manage projects with platforms like Asana and monday.com, and build charts using Hex or Amplitude—all without leaving the chat interface. This feature mirrors other “mini” app integrations seen in messaging platforms like Telegram and Discord. The broader vision is to evolve AI platforms into multi-functional ecosystems, much like Tencent’s WeChat. MCP, originally developed by Anthropic in 2024 and donated to the Linux Foundation, is enabling these integrations and now supports interactive app interfaces across different AI environments. The move also coincides with the establishment of the Agentic AI Foundation by leading tech firms to promote open-source development in agentic AI technologies.
AI companies want a new internet – and they think they’ve found the key
AI companies are converging around a new internet protocol, the Model Context Protocol (MCP), to accelerate the development and functionality of AI agents across platforms. Originally developed by two Anthropic engineers in 2024 to improve their chatbot Claude’s utility, MCP enables AI models to seamlessly integrate with apps and services by defining accessible tools and workflows, similar to how APIs fueled the Web 2.0 era. Now widely adopted by OpenAI, Microsoft, Google, and others, MCP is becoming a de facto standard for interoperable AI systems. To ensure its neutrality and foster broader industry collaboration, Anthropic has donated MCP to the Linux Foundation. This move, along with contributions from OpenAI and Block, supports the formation of the Agentic AI Foundation (AAIF), focused on open-source agentic AI. This open governance model addresses security concerns and encourages further development. MCP aims to streamline AI task execution by allowing agents to operate directly through services, improving speed, accuracy, and reliability. Its broader adoption could revolutionize user experiences, shifting internet use from browser-based actions to AI-enabled interactions. While the future of MCP is not guaranteed, it sets a foundation for standardizing AI communication and task orchestration.
Cloudflare CEO Matthew Prince wants Google to block AI crawlers without affecting search results; says ‘We will get Google to…’
Cloudflare CEO Matthew Prince has urged Google to offer more precise controls to block AI crawlers like its chatbot Gemini, without interfering with normal search engine indexing. In a discussion on X (formerly Twitter), Prince revealed that Gemini is already blocked within Cloudflare’s systems. He stressed the importance of cooperation from major AI companies to combat unauthorized web scraping. Cloudflare recently launched a “pay-per-crawl” model aimed at countering AI firms that extract data from the open web without compensating content owners. Prince asserted that the companies conducting the crawling should be the ones to support such regulatory and monetization initiatives.
Anthropic and OpenAI are crawling the web even more and not giving much back
The article from Business Insider examines how major AI companies like Anthropic and OpenAI are increasingly crawling the web for data while providing diminishing returns to the sites they extract it from. Based on new data from Cloudflare, which tracks website crawl and referral activity, these companies exhibit high “crawl-to-refer” ratios—indicating heavy content scraping with minimal user redirection to source websites. This undermines the traditional “grand bargain” of the internet, where websites provided data in exchange for traffic and monetization opportunities. As generative AI engines deliver direct answers to users, fewer people visit original content sources, further devaluing web publishers’ efforts. Some site owners are even experiencing increased cloud costs due to excessive bot traffic. While companies like Google maintain lower ratios due to traditional search behavior, the broader trend raises concerns over the ethical implications of how AI models are built and the web ecosystem’s sustainability. Despite requests for comments, companies like Anthropic and OpenAI did not respond or questioned the methodology. Business Insider plans to continue monitoring this evolving issue.
Introducing the Model Context Protocol
Nov 25, 2024 — The Model Context Protocol is an open standard that enables developers to build secure, two-way connections between their data sources and AI-powered tools.Read more
Code execution with MCP: building more efficient AI agents
Nov 4, 2025 — The Model Context Protocol (MCP) is an open standard for connecting AI agents to external systems. Connecting agents to tools and data …Read more
Building agents with the Claude Agent SDK
Sep 29, 2025 — In this post, we’ll highlight why we built the Claude Agent SDK, how to build your own agents with it, and share the best practices that have …Read more
Model Context Protocol
The Model Context Protocol (MCP) is an open protocol that enables seamless integration between LLM applications and external data sources and tools.Read more
modelcontextprotocol/servers: Model Context Protocol …
This repository is a collection of reference implementations for the Model Context Protocol (MCP), as well as references to community built servers and …
modelcontextprotocol/inspector: Visual testing tool for MCP …
Visual testing tool for MCP servers. Contribute to modelcontextprotocol/inspector development by creating an account on GitHub.
modelcontextprotocol/python-sdk
The Model Context Protocol (MCP) lets you build servers that expose data and functionality to LLM applications in a secure, standardized way.
Overview of OpenAI Crawlers
OpenAI uses web crawlers (“robots”) and user agents to perform actions for its products, either automatically or triggered by user request.Read more
Specification
Nov 25, 2025 — Model Context Protocol (MCP) is an open protocol that enables seamless integration between LLM applications and external data sources and tools.Read more
What is the Model Context Protocol (MCP)? – Model Context …
MCP (Model Context Protocol) is an open-source standard for connecting AI applications to external systems. Using MCP, AI applications like Claude or …Read more
Claude User Agents
Learn about Anthropic Claude crawlers and user agents, how they access your website, and how to optimize for Claude AI.
OpenAI Crawlers and User Agents Guide
A complete guide to all OpenAI crawlers and user agents, including GPTBot, ChatGPT-User, and OAI-SearchBot, and how they interact with your website.
GPTBot makes over 10000 request to my website – Bugs
Apr 23, 2025 — GPTBot user agent information is published at https://platform.openai.com/docs/bots, which you can use to restrict crawling in your robots.Read more
Does Anthropic crawl data from the web, and how can site …
Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent. Claude-User allows site owners …Read more
Understanding OpenAI’s GPTBot & Robots.txt Setup
Utilize OpenAI’s documented user agent token “GPTBot” and complete the user-agent string for precise identification. Incorporate “Disallow” & ‘Allow’ rules …Read more
ClaudeBot User Agent – Anthropic Bot Details | CL SEO
ClaudeBot is Anthropic’s web crawler used to collect training data for Claude AI models. Operating similarly to other AI training crawlers, ClaudeBot …
What is MCP? The Universal Connector for AI Explained
Sep 5, 2025 — At its core, MCP is an open protocol introduced by Anthropic in late 2024. Its fundamental purpose is to enable AI models to securely connect to …Read more
GPTBot: OpenAI’s Web Crawler
GPTBot is OpenAI’s proprietary web crawler. Its primary purpose is to crawl web pages to potentially improve future AI models.Read more
What Is ClaudeBot User Agent? A Complete Guide to …
Jan 11, 2026 — Learn what ClaudeBot user agent is, how Anthropic’s AI crawler works, why it visits websites, and how to manage it through robots.txt for …
Securing the Model Context Protocol (MCP): A Deep Dive …
Jun 20, 2025 — Originally proposed by Anthropic, MCP has quickly become the de facto open standard for allowing language models to securely interact with …Read more
How to block bots from OpenAI GPTBot
Nov 10, 2025 — To block OpenAI GPTBot specifically, you can start with the “Block AI Bots Firewall Rule” template and modify it for the defined user agent.Read more
What Is MCP (Model Context Protocol)?
Model Context Protocol (MCP) is an open standard introduced by Anthropic in November 2024 to streamline how large language models (LLMs) connect to external …Read more
What is GPTBot? – Hall: AI
GPTBot is OpenAI’s web crawler that systematically browses and collects publicly available internet content to train and improve their large language models …
Getting hammered by some bot – Getting Started
Jun 21, 2024 — Create a new custom firewall rule. Set the rule to match the user-agent header containing “ClaudeBot.” Set the action to “Block” for this rule.Read more
What is model context protocol (MCP)?
Model Context Protocol (MCP) is an open-source AI standard developed by Anthropic, released in November 2024, to facilitate secure, real-time interaction between large language models (LLMs) and external data sources, tools, and services. Traditionally, LLMs were siloed to their training data, requiring manual, custom-coded integrations for access to real-time external data. MCP resolves this issue through a standardized client-server architecture, enabling simplified and scalable connections via STDIO, HTTP, and SSE. MCP includes three components: the host (LLM-embedded application), the client (which formats user requests), and the server (which executes and returns those requests). MCP reduces the integration challenges of the N x M problem (many models × many tools) by offering a reusable, pluggable protocol suited for cloud-native, composable systems like MACH architecture. It has seen broad adoption across the AI and tech ecosystem, including OpenAI, Google DeepMind, Microsoft, and startups. Use cases span cloud management, multi-agent orchestration, software development, and personalized assistants. However, MCP poses security risks due to its reliance on trust. Concerns include prompt injection, incorrect server implementation, and a new attack vector called MCP-UPD (Unintended Privacy Disclosure), where attackers embed malicious instructions in data. Experts emphasize strong authentication measures as MCP evolves.
Hot new protocol glues together AI and apps
A new technical standard called the Model Context Protocol (MCP), developed by