How AI Is Disrupting the Internet's Advertising Economy

# How AI Is Breaking the Internet Economy

Written By Gabe Tramble @zaddycoin and Paul Timofeev @ptimfv

Special thanks to Rohin Lohe at Cloudflare, Jennifer Ong and Kyle Corcoran at TollBit, Tim Conard and Cuy Sheffield at Visa, and Son Do at Parallel.ai, for feedback and contributions. (https://x.com/rohinlohe) (https://www.linkedin.com/in/kylecorcoran47/) (https://www.linkedin.com/in/tim-conard-a51458115) (https://x.com/cuysheffield) (https://www.linkedin.com/in/sqdo) (https://Parallel.ai)

On March 19th, 2026, Matthew Prince, the CEO of Cloudflare, the world’s largest content delivery network sitting at ~20% of the web, predicted that bot traffic, driven by the rise of generative AI, would exceed human traffic in 2027. Just a few months later, Prince announced that agentic traffic had grown so quickly that bots had already overtaken human traffic. Bots account for roughly 60% of HTTP requests for HTML content across Cloudflare’s network, compared with 38.2% from humans. As automated systems generate a larger share of HTML page requests than humans and AI agents account for a growing share of that machine traffic, the underlying business model of the internet is breaking. (https://x.com/eastdakota) (https://x.com/Cloudflare) (https://x.com/eastdakota/status/2062212701414187452?s=46) (https://radar.cloudflare.com/)

## The Bargain That Funded the Open Web

The world wide web has enabled unprecedented global democratization of information, a mission which has been funded by a straightforward content-for-traffic barter exchange, wherein publishers would publish content on the web, and search engines would direct readers to their sites if it matched what they were looking for. Publishers provided information or entertainment to the web in the form of content; search engines helped direct readers to relevant webpages; and advertisers would monetize the user visit by displaying targeted ads on the webpages users were led to. Together, this synchronized loop established the multi-billion dollar advertising machine that funded the open, paywall-free web.

The exchange funded a vast amount of freely accessible information while concentrating distribution power in search and advertising platforms. It rewarded behavioral advertising, SEO-optimized volume, and engagement-driven content, leaving publishers dependent on referral and attention-based traffic. AI is chipping away at the referral leg from an already fragile bargain before a replacement was ready.

Google's advertising revenue grew from $66.9 million in 2001, its first fully disclosed advertising year, to $294.7 billion in 2025, a factor of about 4,400.

The explosive growth of AI chatbot applications in recent years, however, has introduced a new paradigm for discovering information on the web. By design, these AI apps prioritize user convenience by keeping the audience within the chat interface itself. Though convenient for users, this feature has become particularly detrimental for publishers. While search engines directed readers to websites where the content they’re looking for originates from, AI applications instead condense and summarize the information and present it as part of their final response to a given user prompt. It becomes harder for publishers to monetize web traffic as they lose control of the direct user relationship and start to receive less human web traffic.

One emerging solution is to introduce new "pay-per-request" payments infrastructure for content and search on the internet. In this paper, we explore how AI is “breaking the internet”, the two distinct approaches to this solution, and what the state of the newly emerging pay-per-request web looks like today.

## AI Crawlers And The Death Of The Barter

A search engine uses a crawler, an automated program designed to index webpages,  to build a ‘map’ of the web. Under the traditional content-for-traffic model, publishers allowed these crawlers to copy their text for free because the search engine rewarded them by routing human readers back to their websites. To manage this access, the industry relied on a text file called robots.txt, an unencrypted, voluntary honor system. Robots.txt worked smoothly for as long as it did because defying robots.txt meant losing access to publishers’ content, which negatively impacts the search engine's index of the web.

Generative AI applications disrupt the web’s content-for-traffic barter exchange by turning crawlers from indexers into active consumers, which unfolds most evidently in training and real-time retrieval. During training, a large language model like GPT or Claude learns from a massive corpus of data scraped from the open web, which includes various books, articles, code, and art. The model trains on the totality of this human knowledge and serves it back to users with no link to the source and no payment to the creator. When retrieving information on the web in real-time to answer a user prompt, the model indexes the live web at the exact moment a user asks a question, pulls the relevant passage, and reformats it. Rather than handing the reader the original page, the model restates the information into a fresh, inline reply that answers the prompt entirely. A crawler indexes a page once, and the model can serve what it found to millions of readers, none of them ever carrying the reader back to the source.

The Impact

When taking the mechanics of extraction, alongside behavioral data, federal legal battles, and artistic exploitation, it becomes evident that the open web's foundation is experiencing a structural collapse, as the impact is already showing up across several interconnected domains.

The behavioral impact of AI crawlers can be measured by the ‘crawl-to-refer’ ratio: the number of pages a crawler scrapes versus the number of actual users it sends back. Data from Cloudflare shows that in the past six months alone, both training (43.9%) and training and search (41.5%) drove over 85% of identified AI crawler requests observed across Cloudflare’s network.Googlebot still runs nearly five pages for every visitor it sends back, 5.3 over the past year, which is more in line with historical exchanges, and every major AI platform sits one to three orders of magnitude above it. Over the past year, Anthropic’s ClaudeBot averaged 4.2k page crawls for every single visitor sent back to an origin site, Mistral 884, OpenAI 731, and Perplexity 147. Cloudflare does note that referrals from native apps don’t carry referral headers, so these ratios could overstate the imbalance. A Pew Research Center study found that when AI summaries intercept search queries, human click-through rates drop from 15% to 8%, direct click attribution with summaries drops to 1%, and users abandon browsing sessions entirely 26% of the time. (https://x.com/AnthropicAI) (https://x.com/MistralAI) (https://x.com/perplexity_ai) (https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/)

AI crawlers have also prompted legal action. In Bartz v. Anthropic, Judge William Alsup ruled that training on lawfully acquired books was fair use because the use was transformative, meaning it created something new rather than substituting for the books themselves, while the claim over the pirated libraries Anthropic had downloaded moved toward trial. Anthropic settled that claim privately for $1.5 billion, roughly $3,000 per work across some 500,000 titles. The outcome was widely read as a win for Anthropic, because the fair-use ruling on training stood and the settlement closed off statutory damages that could have run far higher. As the case settled, no court ever established what training on pirated content actually costs. Meanwhile the New York Times v. OpenAI and Microsoft case is still ongoing, forcing OpenAI to turn over 20 million ChatGPT conversation logs as part of an investigation into how closely models reproduce articles to substitute for paid subscriptions. (https://copyrightalliance.org/wp-content/uploads/2025/06/Bartz-v.-Anthropic-Order.pdf) (https://arstechnica.com/tech-policy/2026/06/microsoft-built-supercomputer-to-help-openai-infringe-copyrights-nyt-alleged/)

These cases highlight that the costs of the unmetered extraction of AI crawlers falls on independent creators and visual artists in particular. When OpenAI released its Studio Ghibli-style rendering capabilities in early 2025, it highlighted just how thin existing copyright protections are. Because U.S. law protects specific expressions of a work rather than its overall style, AI models can systematically clone an artistic aesthetic built over decades of human labor without naming or compensating the creators. OpenAI's policy blocks prompts for a living artist like Hayao Miyazaki, yet completely permits prompts for "Studio Ghibli”.  A specific character like Totoro is protected, but the general Ghibli style is not, which leaves the artist with little legal ground even though the model could produce that look only because it trained on the studio's films. Pay-per-request infrastructure cannot resolve every harm described above. It can meter and compensate for future access to a work, but it cannot retroactively compensate artists for material already used in training or prevent a model from reproducing a style it has already learned. Those harms require separate licensing, copyright, and enforcement remedies. (https://x.com/OpenAI)

In short, the two prominent responses to the breakdown of the content-for-traffic barter, litigation and private licensing, have been temporary stopgaps rather than scalable business models. Litigation has so far priced past scraping while leaving ongoing and future crawling untouched, and private bilateral licensing deals let large media conglomerates sell access to their archives directly to AI labs (e.g. OpenAI’s $250m contract with NewsCorp, or Google’s reported $60m-a-year deal with Reddit). Though expensive, a market survey found that most of these private deals amount to less than one percent of the licensee’s projected AI revenue. But negotiating such deals requires extensive financial and legal resources, which means the independent long-tail of the web, the millions of independent content creators, still have no way to deal with AI crawlers en masse.

It then becomes clear that the unencrypted honor system of robots.txt and the barter content-for-traffic exchange that supported the open web for thirty years, cannot withstand an internet economically optimized for data consumption over referral traffic, where crawlers are first-class consumers. Whereas in the past, publishers were happy to bear 100% of content production costs given they would capture 100% of consumer traffic and attention; generative AI consumes the content entirely with little to no attribution, effectively starving the producer of the human web traffic needed to generate ad or subscription revenue and leaving publishers to foot expensive hosting bills for bot scraping while receiving zero traffic. The squeeze runs in both directions at once. Serving a bot costs the same bandwidth and compute as serving a reader, but the bot leaves without viewing an ad or starting a subscription. At crawl-to-refer ratios in the hundreds or thousands, a publisher pays to serve thousands of requests for every visit it can actually monetize. Because publishers cannot afford to pay for servers just to have bots copy their text for free, the internet needs a new way to ensure creators get paid for what they produce. Since people are reading AI summaries instead of visiting websites, publishers can no longer make money from human attention alone. As a result, the point of monetization becomes the moment the AI bot actually scrapes the data, and the next logical step that follows is to introduce new web-native pay-per-request payment infrastructure.

## The Pay-Per-Request Web

Scaling a pay-per-request model across the entire web requires a way to efficiently meter access to content and servers. While an increasing share of this activity is being driven by AI crawlers, the framework extends to any non-human interaction, including data feeds and automated APIs. Put simply, to access paid content, one piece of software must programmatically pay another piece of software, a process formally recognized as machine payments. (https://radar.cloudflare.com/ai-insights)

To support machine payments at scale, infrastructure companies build automated billing gates at the network routing layer to charge a bot at the millisecond of request. However, execution has mostly been divided into two distinct approaches, the gated marketplace, which requires registering a crawler for an account and a unique digital key to access a site through an intermediary routing proxy; and the open protocol model, designed around using standardized public software rules, machine-readable headers, and supports payment settlement across a wider range of digital rails.

The Gated Marketplace

The gated marketplace model treats web content as a private asset managed through a corporate intermediary. Under this framework, identity verification and access clearance are required before the machine payment itself. The publisher configures their network to redirect incoming AI crawlers away from the public, human-facing website and toward a secondary, dedicated web address managed by the marketplace provider. At this gateway, the platform checks the crawler's identity credentials against a database, evaluates the publisher's custom access rules, tracks usage, and formats the page into machine-readable text. Financial transactions occur outside of the internet protocol layer. When a bot makes a request, the marketplace logs the usage and meters it against a credit balance the AI company has funded in advance. Settlement then happens periodically in fiat through merchant platforms like Stripe Connect, since charging a card for every request would cost more in card fees than the request itself. Having an intermediary directly between the AI crawler and the publisher's origin server helps maximize security, shield the server from aggressive traffic spikes, and guarantee that only verified corporate partners gain entry, however a framework like this implemented at scale risks turning the open web into a network of permissioned storefronts, only further deteriorating the human user experience. (https://x.com/stripe)

The initial wave of programmatic pay-per-view payment infrastructure for the web have been gated marketplaces, most notably that of TollBit and the first iteration of Cloudflare’s Pay-Per-Crawl. TollBit is a platform for managing, monitoring and monetizing AI agent web traffic; to deploy it a publisher sets up a specialized subdomain (e.g. https://tollbit.thedailydispatching.com/), and when an AI crawler has been detected attempting to access the main site, the server automatically redirects the crawler to TollBit’s subdomain where the crawler can only gain access upon paying. TollBit checks if the AI company has a registered billing account, logs how many pages the bot requests, and delivers the data in clean text formatting while processing the payments in US dollars. This requires an API key, so discovering the cost of content is not self-serve today: an AI developer has to register with TollBit first. (https://x.com/TollbitOfficial)

On July 1 2025, Cloudflare rolled out its initial Pay-Per-Crawl model. Cloudflare operates at the Domain Name System (DNS) and content delivery network levels, meaning it handles web traffic before it actually reaches a content publisher’s physical origin server. In Pay-Per-Crawl V1, Cloudflare integrated its edge-level Web Application Firewall (WAF) to automatically detect and block known AI crawlers across its network by default. When an AI crawler would request access to a domain protected by Pay-Per-Crawl, Cloudflare would intercept the request before it could reach the publisher’s webpage, and facilitate payment if the site owner chose to monetize its content. (https://blog.cloudflare.com/introducing-pay-per-crawl/)

Challenges With The Gated Marketplace

TollBit relies on developer API setups, proprietary keys, and traditional banking rails for payments. Routing pay-per-request transactions through merchant networks like Stripe introduces fixed transaction processing fees (typically around thirty cents plus a percentage of the total charge) which makes an individual, fraction-of-a-cent request in real time economically unviable. To circumvent this cost barrier, human developers need to manually register an account with the gated marketplace intermediary, generate unique API keys, and pre-fund a balance with a credit card or bank transfer from which the platform deducts fees over time. As such, agents and other autonomous software can’t yet independently discover and purchase data on demand; rather human administrators need to manually establish commercial accounts, monitor fluctuating balances across separate providers, and coordinate security keys.

Though keeping a human in-the-loop is an essential part of many AI-centric processes, using a gated marketplace model to manage access to web content at scale assumes predictable, linear web traffic patterns, which directly conflicts with the operational realities of modern software. AI agents and crawlers use reasoning loops to choose their own navigation paths dynamically, which depend on their given user prompt, meaning they frequently encounter unfamiliar domains containing the precise data needed for a task; stopping a task midway to require manual human onboarding can quickly bottleneck execution and consequently ruin the user experience. Flat-rate subscriptions are not the most efficient solution here either; AI agents operate whenever directed to, which often comes in unpredictable, concentrated bursts (e.g. someone may work on a research-intensive project for 3 days, then take a break for the next 2 weeks).

Between the upfront capital deposits and custom software integrations required to access protected content and data, the gated marketplace functions as a highly permissioned system. On one hand, web publishers receive controlled, verified traffic from established technology firms; on the other, the administrative and financial burden of managing pre-funded accounts across separate platforms restricts non-anonymous access, prevents spontaneous scraping by independent developers, and limits monetization opportunities for smaller publishers who lack high-volume enterprise traffic. In contrast, direct machine-to-machine financial handshakes allow publishers to attach a micro-fee to the text and data that would get requested by a crawler; this design concept forms the basis of the open protocol model.

## The Open Alternative: x402 and MPP

An open protocol is a network architecture built on public, standardized communication rules. Much of the modern web was built on open protocols including HTTP, SMTP, and TCP/IP, which it still uses to this day. In recent years, the HTTP 402 status code, which had been reserved but inactive for over thirty years, has become a core web-native infrastructure component for supporting machine payments on open-source protocols using blockchain rails. (https://shoal.xyz/research/writing/http-402-how-machines-move-money/)

Bitcoin payments using HTTP 402 have been experimented with as far back as 2015, pioneering efforts including the 21co Bitcoin Computer and Lightning Network’s L402. In May 2025, the x402 protocol introduced stablecoins into the HTTP 402 flow for the first time, and efforts like Stripe x Tempo’s Machine Payments Protocol (MPP) support Bitcoin and credit card payments alongside stablecoins as well. The card option is designed for macrotransactions rather than micropayments, because fixed card fees swamp a fraction-of-a-cent request. Stablecoins remain the rail that makes pay-per-request pricing workable. (https://x.com/tempo)

x402 embeds payment instructions directly into the data exchange code, eliminating the need for third-party account registration or proprietary clearinghouses. When an AI crawler requests a webpage, the host server natively responds with a  machine-readable (payload header) embedded straight inside the connection metadata. The header provides the exact price per request, the designated payment network, and the receiving digital billing address.

Blockchains are playing an increasingly prominent role in facilitating machine payments because they provide an immutable, programmatic financial layer that operates at the same native speed and automation level as software logic. Crypto assets like stablecoins sit structurally close to the web as AI agents because they live through the same web-native infrastructure that AI agents and crawlers do, existing as software code that can be integrated via simple API endpoints and cryptographic signatures. Costs and fees for payments using blockchain rails remain significantly cheaper, often fractions of a cent per transfer on leading high-throughput chains, than credit cards and fintechs (though there are a growing number of fintechs actively seeking to close that gap, e.g. Wise). (https://btsesolutions.com/articles/crypto-vs-traditional-payments-2026) (https://owners.wise.com/news-releases/news-release-details/wise-group-plc-reports-full-year-2026-financial-results)

Blockchain rails also provide cryptographic receipts and verification for each payment by design. Each pay-per-request loop provides a permanently stored receipt containing the buyer’s digital signature, the requested URL, and the exact timestamp. Because these instructions are stored on a public chain, any automated agent can parse the incoming metadata, execute a micro-payment using compatible assets (such as stablecoins), and download the content in a single continuous HTTP handshake. And whereas settlement on traditional banking rails requires several days to reconcile debits and credits between commercial bank ledgers via clearinghouses, blockchain settlement occurs as soon as a network’s nodes verify the digital signature and update the global chain state, which as mentioned earlier often occurs in seconds on leading chains. The open protocol approach, above all, standardizes web data access, allowing publishers of any size to monetize their assets and allowing any developer with a funded automated account to crawl without asking for prior administrative permission.

On June 15, 2026, Amazon Web Services (AWS) integrated the x402 protocol directly into CloudFront and its Web Application Firewall. This configuration allows roughly a quarter of the internet to activate automated agent billing straight from their existing cloud dashboards, delivering content to paying bots. Cloudflare followed suit: while its Pay-Per-Crawl V1 required Cloudflare to handle payments as the Merchant of Record through Stripe, the newer Cloudflare Monetization Gateway removes Cloudflare from the payment loop entirely, as payments route peer-to-peer into the seller's wallet. (https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization/)

The primary difference between the gated marketplace and the open protocol comes down to who is allowed to purchase access to web content. Under the gated marketplace model, access is restricted to a curated whitelist; an AI bot cannot buy content unless its parent corporation has manually signed a corporate contract and completed a background check with the intermediary platform. Under the open protocol model, access is entirely automated and unrestricted; any software program on the internet can instantly buy and download data, provided its automated billing account can settle the requested micro-fee on the spot. As both systems roll out across the internet's routing networks, publishers face a choice between two distinct infrastructure options. The gated marketplace model caters to enterprise organizations that prioritize strict security compliance, legal indemnification, and curated corporate partnerships. Conversely, the open protocol model serves automated workflows that require frictionless micro-transactions, allowing a diverse ecosystem of independent developers and publishers to exchange data programmatically without administrative hurdles.

What AI Crawlers Actually Encounter Today

To measure the actual footprint of the pay-per-request web, Shoal Research probed 108 major publishers and consumer websites as an AI crawler (GPTBot) on August 19, 2026. The sample has since been expanded from 108 to 500 domains. The results below describe the original August run.  For each request, we recorded status codes, redirect chains, response headers, and challenge bodies, computing SHA-256 hashes to guarantee data integrity across collection runs. An endpoint was classified as x402-payable or agent payable if and only if its HTTP 402 response returned parseable payment parameters within a Payment-Required / X-Payment-Required header or a valid x402 JSON challenge payload. (https://github.com/shoalresearch/x402-publisher-study)

Across the 108 publishers, the web divided into three main postures:

- Served for Free (49.1%, n=53): 53 domains served their homepages openly with an HTTP 200 OK, allowing unmetered extraction.

- Blocked at Perimeter (33.3%, n=36 combined): 31 domains returned HTTP 403 Forbidden, two required authentication (HTTP 401), two rejected content negotiation (HTTP 406), and one returned a legal block (HTTP 451). Three additional publishers timed out after repeated attempts.

- Returned HTTP 402 (14.8%, n=16): Exactly 16 publishers returned an HTTP 402 Payment Required status code.

Crucially, not a single publisher in the dataset returned an open, machine-payable challenge.

Of the 16 HTTP 402 responses, 10 redirected to a TollBit Bot Paywall, and six returned static HTML licensing contact forms. A secondary check using authenticated developer keys across all 13 TollBit-managed publishers in the baseline dataset returned HTTP 403 on every single rate lookup. The API payload explicitly stated that the content provider had disabled programmatic access and no price-per-scrape rate was returned.

One way to interpret these results is that TollBit is serving as an early AI partner for traditional publishers to test and understand pay-per-use billing for content, while still giving publishers the control and safety of managed access to agents. A system that requires manual developer registration, proprietary API keys, and corporate whitelisting is difficult to scale to an unprecedented amount of autonomous agents navigating dynamic web paths; rather it is an interim measure designed to protect legacy revenues while a permanent, machine-native settlement standard takes shape. It's early, and we think providers like TollBit are looking to understand, and formulate, how to balance the access, licensing, and autonomy that comes with this type of service.

We also pulled the complete Coinbase CDP registry on August 19 to map out who is actually using x402 and what services are live on the network which boasts over 15k endpoints where a few hundred are actively used. (https://x.com/coinbase)

Mainstream content publishers have not adopted x402 yet, but beneath that absence, a bustling supply-side market is already active. While consumer media is not quite actively integrated yet, developer tools, specialized data vendors, and proxy networks have rapidly deployed open machine-payment endpoints; the recognizable names from the x402 ecosystem include data APIs such as Tavily, Nansen, CoinGecko, Chainlink, Apify, and Browserbase, and more recently Ramp. Top service providers on x402 include names like ChatGPT, Claude, TripAdvisor, and Perplexity, but no major content publishers yet. (https://x.com/tavilyai) (https://x.com/nansen_ai) (https://x.com/coingecko) (https://x.com/chainlink) (https://blog.apify.com/introducing-x402-agentic-payments/) (https://www.browserbase.com/blog/browserbase-and-coinbase-x402) (https://x.com/teddy_riker/status/2090467979121013077?s=20) (https://agentic.market/)

15,073 active endpoints suggest that software is actively pricing and selling programmatic access to other software on open public rails. Some institutions tracking the network report up to 50,000 endpoints once bundled marketplaces are included. However, the endpoints remain strictly specialized around scraping, AI tool-calling, and live data feeds; when cross-referencing this entire registry against mainstream consumer media and publishing domains (The New York Times, Reuters, ESPN, Wikipedia), the match rate was zero, which we expect to change slowly but surely  as typically, it takes one big name to “get the ball rolling”. Web monetization shifts historically follow a single incumbent breaking “the status quo”; The New York Times rolled out the metered paywall in 2011; the Associated Press set the precedent for commercial AI data deals in 2023 that triggered a cascade from News Corp to TIME to Condé Nast; therefore we expect that when the first tier-one publisher prices its access on an open protocol (e.g. x402, MPP) we will see the rest of the market waste little time to follow suit.

Challenges With The Open Protocol Approach

The infrastructure for open pay-per-request protocols is operational, but the market itself remains nascent. The technical plumbing for open payments is actively being deployed: publishers have machine-readable rights frameworks like RSL, hosting and routing gates in AWS CloudFront and Cloudflare that can charge at the millisecond of request, and settlement rails from credit cards to digital assets and stablecoins, all universally accessible through protocols like x402 and MPP.  The largest remaining hurdles for active adoption rest entirely on the demand side.

For starters, no major AI lab routes production crawler traffic through open machine payment rails for consumer web content, for which we see three main reasons. First, crawlers lack an integrated wallet, automated procurement budgets, and most importantly, a commercial obligation to actually pay for the content it is requesting. As shown by our crawler probe, much of the web is still completely ungated, which means an agent blocked by HTTP 402 can simply drop the URL and extract near-or-entirely-identical content from an open ungated competitor at zero cost. The legal environment reinforces this dynamic: under federal rulings like Judge William Alsup's decision in Bartz v. Anthropic, the courts established that ingesting lawfully accessible works to train an AI model is transformative fair use. While Alsup penalized the retention of illicitly acquired pirate datasets, training on content accessible across the open web carries zero copyright liability. Because extracting public web text is legally free, AI labs have no commercial incentive to voluntarily pay open micro-fees.

How AI Search Providers Are Approaching Compensation

Production adoption remains limited, but AI search providers are beginning to treat compensation as part of product design.

In an unpublished Shoal Signal interview, Son Do, a forward deployed engineer at Parallel, described Index, an initiative that works directly with publishers, measures how much each source contributes to an agent's final answer, and shares revenue with those sources. Do expects publishers and other data providers to move toward usage-based models, with agentic payment rails supporting large numbers of small transactions. Index shows that compensation is entering the design of AI search and retrieval products before it has become standard purchasing behavior across major AI labs. (https://www.linkedin.com/in/sqdo) (https://x.com/p0) (https://parallel.ai/blog/introducing-index-by-parallel)

Beyond the legal and economic lack of obligation, existing open payment protocols suffer from a simple structural flaw that is, metering the wrong part of the pay-per-request loop, metering the initial HTTP crawl rather than downstream inference, when the model synthesizes ingested knowledge to answer a user prompt. A sub-cent fee to fetch raw HTML does not compensate a creator if an AI model can absorb that reporting once and reproduce it across millions of user answers. While frameworks like RSL define pay-per-inference options, network-level gates like Cloudflare or AWS cannot inspect the latent weights of a neural network to cryptographically verify which training documents generated a specific output token. Cloudflare’s July 2026 pilots with Ceramic.ai and You.com highlight this exact technical bottleneck. The initiative attempts to move away from charging per crawl, testing a 'pay-per-query' model that compensates publishers only when their text is actually cited in a user-facing AI answer. While this correctly targets the moment where value is created, network edge providers like Cloudflare sit outside the AI's servers, meaning Cloudflare cannot independently verify which sources a model referenced during generation, and thus the system has to rely on the AI company counting its own citations and self-reporting how much it owes, reverting back into an honor system.

Another fundamental challenge to the open protocols is the fact that you can’t really stop an AI crawler from paying a microfee for content once, then writing the text to its internal cache or memory, and providing that data indefinitely. You also can’t stop an AI crawler from doing this thousands of times over and over across different websites and publishers. As Brendan Ryan, an MPP co-author at Tempo, argued in a Shoal Signal interview we conducted, a seller’s ability to charge depends heavily on whether its information is unique, current, and difficult to cache. (https://x.com/brendan_j_ryan) (https://shoal.xyz/research/episodes/tempo-mpp-agent-payments/)

This limitation is most acute for the static creative work and reporting discussed earlier. Static text loses its economic value almost immediately after it is published, even if collected using retrieval mechanisms from AI tooling. Once an AI model reads an investigative piece, historical essay, or news article, that information is absorbed into the model's memory. The model can answer questions using those facts forever without ever visiting the original website again. Because of this, software tools that simply package public web text with an AI interface will see their profits disappear quickly, as rivals build identical scrapers over the same free information.

The underlying rule is then simple: static data loses its pricing power almost immediately, while dynamic and un-scrapable information stays payable. Because an AI model can read an unpaywalled article or historical report once and store that knowledge permanently in its weights, pay-per-request micro-fees cannot sustain businesses that rely on static text; ongoing payments only work for information that must be produced and verified continuously, which we believe comes down to three main categories of content:

First is live, ephemeral state, including real-time order books, sub-second financial pricing, breaking news telemetry, and IoT sensor feeds. An AI model cannot answer queries about the present state of the world from months-old training weights or a stale cache; it must issue a fresh, paid API request to an origin server at that exact second to retrieve current reality.

Second is proprietary offline data that never touches the open web. In private market intelligence, for example, historical funding rounds and executive moves published in press releases are quickly scraped and commoditized by low-cost rivals. The real pricing power belongs to providers that conduct primary, manual research, such as gathering non-public cap table terms, private deal valuations, and verified LP allocations directly through confidential relationships with fund managers. Because this data is never indexed on a public URL, automated crawlers cannot bypass the paywall or extract it for free.

Third is a distinct human voice and brand reputation. As media strategist Ashni Christ pointed out on Shoal Signal, a generic news aggregation feed that gets blocked or goes offline is instantly replaced by an identical automated scraper without any audience loss. However, when an authentic analyst, investigative journalist, or creator with an established viewpoint stops publishing, readers immediately notice and search for them. Live interviews, subjective analysis, and direct access cannot be pre-computed by an LLM, stored in an intermediary cache, or replicated from static training sets, leaving live feeds, private offline records, and personal perspective as the only assets that can reliably command a price. (https://x.com/ashnichrist) (https://x.com/Shoalresearch/status/2051350822332801086?s=20)

## Closing Thoughts

The internet is breaking, or so it appears.

To accelerate communication and coordination with each other, humans built and connected global networks of computers and software. Over the decades, every successive layer of technology aimed to make that exchange faster and more efficient, ultimately culminating in software built for mass automation. Generative AI put that automation power directly into the hands of hundreds of millions of people at once. But as automated software takes over the network, it is fundamentally altering how information is discovered, synthesized, and consumed. The most severe consequence of this shift is economic: it is no longer feasible to monetize web traffic through the same ad-impression and human-referral models that sustained the open web for the past thirty years; hence why we see the pay-per-request web forming now.

The two distinct existing approaches are a tale as old as time on the web: centralized, closed-source intermediaries, who offer fiat billing, turnkey compliance, and familiar enterprise onboarding, but also bring platform lock-in and permissioned gatekeeping; and open protocols, who offer permissionless access and programmable micropayments via stablecoins, though they still face gaps in enterprise distribution, inference attribution, and most importantly, actual publisher adoption.

Testing the state of the pay-per-request web for ourselves demonstrated that the infrastructure and supply side for open payment protocols is there. Open protocols like x402 are already actively facilitating functional markets for data collection, tool calling, AI inference, and live data feeds, but are still not supported by major online publishers. Platforms like TollBit currently are marketed as programmatic monetization marketplaces, but the data suggests publishers are using them as temporary solutions to block scrapers and funnel AI labs to private offline deals.

Which tells us that deciding whether it is gated marketplaces or open protocols that “win” is far from over. In fact, who even says it needs to favor one or the other? The future will most likely contain both approaches, serving different customers and different use cases. Stablecoins will be just as actively used as prepaid credits, subscriptions, and traditional invoices. The game is nowhere near as zero-sum as some may believe, or at least claim to believe. At least it won’t be for a while.

However, though the internet is at an inflection point and it’s tempting to make forward-looking predictions about where we go from here, it’s important to zoom out and remember the web has only been around for 30 years, and it has evolved continuously over those years. Whether the emerging pay-per-request web converges on gated marketplaces or open protocols for payment infrastructure, the possibility that further developments in the actively evolving, well-capitalized sector of AI render both approaches useless and instead lay the groundwork for a new, unexplored approach always remains. For now, we believe it is more likely that we see growth in adoption across open protocols and gated marketplaces simultaneously, serving different segments and different use cases. As with any forward-looking thesis, however, only time will tell.

The collapse of the old bargain may ultimately create a healthier web, at the same time, could increase the cost of information access for people worldwide who don't have the funds for pay-per-use or agents in the first place However, direct machine payments could give publishers a way to earn from machine use without relying entirely on advertising or platform-controlled referrals. The outcome will depend on whether these new rails remain open to independent publishers or become another layer of gatekeepers.

# References

Amazon Web Services. (2026, June 15). AWS WAF announces AI traffic monetization. https://aws.amazon.com/about-aws/whats-new/2026/06/aws-waf-ai-traffic-monetization/

Attridge, M. (2026, May 14). Authors, publishers near final approval of $1.5 billion Anthropic copyright settlement. Courthouse News Service. https://www.courthousenews.com/authors-publishers-near-final-approval-of-1-5-billion-anthropic-copyright-settlement/

Bartz v. Anthropic, Order on fair use (N.D. Cal. June 2025). https://copyrightalliance.org/wp-content/uploads/2025/06/Bartz-v.-Anthropic-Order.pdf

Bloomberg Law. (2026, January 5). OpenAI must turn over 20 million ChatGPT logs, judge affirms. https://news.bloomberglaw.com/ip-law/openai-must-turn-over-20-million-chatgpt-logs-judge-affirms

Chainalysis. (2026, June 3). Inside x402: 100M agentic payments on Base. https://www.chainalysis.com/blog/x402-agentic-payments-adoption/

Chapekis, A., & Lieb, A. (2025, July 22). Google users are less likely to click on links when an AI summary appears in the results. Pew Research Center. https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/

Christ, A. (2026, May 4). New media in the age of AI: TBPN, OpenAI, and media defensibility (Episode 5) Video podcast episode]. In G. Tramble (Host), Shoal Signal. [https://x.com/Shoalresearch/status/2051350822332801086

Cloudflare. (2025a, July 1). Cloudflare just changed how AI crawlers scrape the Internet at large Press release]. [https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/

Cloudflare. (2025b, July 1). Content Independence Day: No AI crawl without compensation. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/

Cloudflare. (2025c, August 29). The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals. https://blog.cloudflare.com/crawlers-click-ai-bots-training/

Cloudflare. (2025d, July 1). Introducing pay per crawl. https://blog.cloudflare.com/introducing-pay-per-crawl/

Cloudflare. (2025e, July 1). Introducing pay per crawl (private beta) Changelog]. [https://developers.cloudflare.com/changelog/post/2025-07-01-pay-per-crawl/

Coinbase. (2026, June 15). Coinbase and AWS let publishers accept agents as customers via x402. https://www.coinbase.com/blog/coinbase-and-aws-let-publishers-accept-agents-as-customers-via-x402

CoinDesk. (2026a, May 5). AI agents are breaking web economics, but Cloudflare says x402 can help. https://www.coindesk.com/tech/2026/05/05/ai-agents-are-breaking-web-economics-but-cloudflare-says-x402-can-help

CoinDesk. (2026b, March 11). Coinbase-backed AI payments protocol wants to fix micropayments but demand is just not there yet. https://www.coindesk.com/markets/2026/03/11/coinbase-backed-ai-payments-protocol-wants-to-fix-micropayment-but-demand-is-just-not-there-yet

Conroy, M. (2026, July 1). Making AI search smarter. Cloudflare. https://blog.cloudflare.com/making-ai-search-smarter/

Fast Company. (2025, March 28). OpenAI's Studio Ghibli-style images renew the debate over AI and copyright. https://www.fastcompany.com/91308222/openais-studio-ghibli-style-images-renew-the-debate-over-ai-and-copyright

The Independent. (2025, March 27). ChatGPT's viral Studio Ghibli-style images highlights AI copyright concerns. https://www.independent.co.uk/news/studio-ghibli-hayao-miyazaki-openai-chatgpt-los-angeles-b2722945.html

Montclair State University, College of Communication and Media. (2025, April). Social media study of the Studio Ghibli AI trend. https://www.montclair.edu/college-of-communication-and-media/wp-content/uploads/sites/20/2025/04/Montclair-Social-Media-Study-Ghibli-AI.pdf

News Corp. (2024, May 22). News Corp and OpenAI sign landmark multi-year global partnership Press release]. [https://investors.newscorp.com/news-releases/news-release-details/news-corp-and-openai-sign-landmark-multi-year-global-partnership

Quartz. (2026, May 27). The price of AI training data, from $5M to $250M. https://qz.com/ai-training-data-pricing-licensing-deals-market-052126

RSL Collective. (2025, December 10). RSL AI Licensing 1.0 now an official industry standard Press release]. [https://rslstandard.org/press/rsl-1-specification-2025

Ryan, B. (2026). Conversation on machine payments and content defensibility [Forthcoming interview]. Shoal Signal.

Shin, L. (2026, March 30). Machine economy 2030. Artemis Research. https://research.artemis.ai/p/machine-economy-2030

Shoal Research. (2026, September 9). x402 and publisher content access, 108 publishers expanded to 500 Data set]. GitHub. [https://github.com/shoalresearch/x402-publisher-study

Stripe. (2026). Introducing the Machine Payments Protocol. https://stripe.com/blog/machine-payments-protocol

Tempo Labs, & Stripe. (2026, March 30). The payment HTTP authentication scheme (Internet-Draft No. draft-httpauth-payment-00). https://github.com/tempoxyz/mpp-specs