Skip to main content
All Articles
Trends

Websites Have Started Blocking AI Shopping Agents. Yours Might Be One of Them.

Amazon blocked Meta's shopping agent. Walmart partnered with it and customers still could not check out. Cloudflare now blocks user-directed agents by default on any page showing an ad, and Google's Universal Cart is already live. Bot protection has quietly become a commerce decision.

Renato Dequcinis 7 min read

Six weeks ago we wrote about Cloudflare’s 15 September change to AI crawler defaults. That article was about crawlers — the bots that collect your pages to train a model or to index them for search. The question it answered was whether you would still be found.

This is the sequel, and the stakes are different. The question this time is whether your customer can buy.

What actually happened

TechCrunch published a piece on 6 October reporting that Amazon began blocking Meta’s Muse agent from its retail site around 21 September, citing Bloomberg. Muse could no longer browse or buy from Amazon’s catalogue.

That part is unsurprising. Amazon has an obvious interest in not letting a competitor’s agent transact on its storefront.

The Walmart part is the interesting one. Walmart is a Muse partner — the partnership was announced at Meta Connect on 23 September — and its customers’ purchases were failing anyway. The apparent cause was not a policy. It was a human-verification button which, when interrupted, fails the check and boots the agent. A Walmart spokesperson said the failures were not intentional.

Nobody decided to block anything. The bot protection simply did what bot protection does.

The same pattern shows up across the responses TechCrunch collected. Delta said it has no partnership enabling third-party agents to book its flights and is evaluating how they could fit, noting any opening would need “security and the customer experience in mind.” United pointed at its Terms of Use, which bar robots and automated processes without prior written permission. Yelp does not permit non-human traffic unless the agent has paid through its data licensing programme. eBay says it does not block all shopping agents but restricts unauthorised ones. Zillow says it allows agents acting for consumers — and acknowledges that its own anti-bot measures can occasionally block some of them.

Zillow’s answer is the honest one, and it is the answer most businesses would have to give if they were asked. We intend to allow them. Our settings sometimes do not.

The Cloudflare default that most owners have not seen

Here is the piece with the widest reach, and the one almost nobody running a small business website knows about.

Cloudflare’s preset for ad-supported domains does three separate things:

  • allows search crawlers
  • refuses AI training
  • blocks user-directed agents on any page showing an ad

The stated reasoning is defensible: an ad indicates the page was built for a person to look at. If an agent is reading it, the advertiser is paying for an impression nobody saw.

Two details matter more than the reasoning. First, there is currently no opt-out directive for agents — only a block. You cannot politely signal “agents welcome here” in the way robots.txt lets you signal crawler preferences. Second, the default reaches new domains, new sites from existing customers, and free-tier accounts. Cloudflare fronts roughly a fifth of the web.

This is the same restructure we covered in September, when Cloudflare split its single “Block AI Bots” switch into three controls. At the time the interesting consequence was the Googlebot trap. Six weeks on, the more expensive consequence is this one.

For context on how fast the underlying traffic is moving: Cloudflare’s July bot report put AI training at 52% of crawler requests as of June, up from 22% in spring 2025. And the asymmetry in what site owners actually block is stark — fewer than 1% of Cloudflare sites block search bots, while 17% restrict AI training.

Almost everyone has thought about training. Almost nobody has thought about agents.

Why this is not the same problem as crawler access

It is worth being precise, because the three bot classes get flattened together constantly and they have genuinely different economics.

A training crawler takes your material to build a model. Blocking it costs you future influence and saves you bandwidth. Reasonable people disagree about the trade.

A search crawler indexes your pages so they can be surfaced later. Blocking it costs you discovery. Almost nobody does this on purpose, which is why the under-1% figure is what it is.

A user-directed agent is acting for a named person who is trying to do something right now — compare three suppliers, check whether you carry a part, book a slot, get a quote. Blocking it does not cost you a citation or an impression. It costs you that specific transaction, from a visitor who arrived with more intent than almost any other traffic you get.

The first two are strategy questions. The third is a checkout question wearing a security costume.

Three standards, and the one already shipping

Sierra and Meta announced the Personal Agent Protocol on 6 October, written up by Bret Taylor and Clay Bavor, with Genesys, Instinct, Rocket, Shopify, Stripe and Walmart named as partners. A v0.1 specification was due later in October.

What it actually specifies is narrower than the announcements suggest. Sessions are built on OAuth. An agent can start as a guest — checking stock, reading a returns policy — and once the customer signs in, the customer chooses whether the agent gets read-only or write access. A session carries across channels, so a pre-login question and a post-login order change belong to one visit. The business chooses how agents reach it: through the website, through APIs using standards such as MCP and OpenAPI, or through the company’s own agent.

Payments are not in the first specification. They are listed as a future extension, alongside push notifications and more granular permissions.

Visa runs a third, the Trusted Agent Protocol, joined by Microsoft, Stripe, Shopify and Worldpay. Stripe appears in all three.

The one that is already live

The protocol that matters most to a merchant is the one that got the least coverage, because it did not launch this month. Google announced the Universal Commerce Protocol on 11 January 2026, describing it as “a new open standard for agentic commerce that works across the entire shopping journey,” co-developed with Shopify, Etsy, Wayfair, Target and Walmart, and compatible with A2A, AP2 and MCP.

It is no longer a plan. Google began rolling out Universal Cart on it around 15 October, in the Gemini app and through AI Mode, drawing on more than 60 billion product listings in its Shopping Graph. Nike, Sephora, Target, Ulta Beauty, Wayfair and a large number of Shopify merchants have signed up. Retailers have to register to take part.

That last sentence is the whole difference between this and the other two. The Sierra and Visa protocols describe how an agent should behave when it arrives at your site. Google’s describes a place your products sit so that an agent never has to arrive at all. One unresolved question, which the reporting has not settled: saved items and cart-abandonment data live in Google’s ecosystem rather than on the retailer’s site. Google’s own guide says merchants remain merchant of record and keep their first-party data. Whether that holds in practice is worth watching rather than assuming.

There is also a regulatory gap nobody has closed: European strong customer authentication rules have no carve-out for software approving payments on a person’s behalf. The protocols are arriving faster than the arrangements they will have to operate under.

None of this is settled, and we are not going to pretend otherwise. But the direction is clear enough that “we will deal with agents when there’s a standard” is a decision to be absent for the next year or two.

What to actually do, this week

This is a thirty-minute job, not a project.

1. Find out what you currently do. Open your CDN or bot-protection settings and find the AI bot controls. If you are on Cloudflare, check whether the three separate controls — search crawlers, AI training, user-directed agents — are set the way you would choose, or the way they arrived. Our guide to choosing a Cloudflare AI crawler setting walks through the options.

2. Work out whether you serve ads. If you do, and you are on a recent Cloudflare setup, assume agents are being blocked on those pages until you have confirmed otherwise.

3. Decide deliberately, by page type. A content site funded by display advertising has a genuine reason to keep blocking. A site where someone might buy, book or request a quote has the opposite interest on those specific pages. The answer does not have to be the same across the whole domain.

4. Test the paths that matter. The Walmart failure was a verification button, not a firewall rule. Walk your own quote form, booking flow or checkout and ask which steps would defeat something operating without hands — an interstitial, a CAPTCHA, a step that only appears after a hover.

5. Check your logs, as always. Your server logs are still the only evidence in this field that no platform controls, no vendor samples and nobody can quietly reprice. If agents are reaching you, they are in there.

The honest summary

We do not know how much agent traffic is reaching South African business websites today. We have not seen credible figures and we are not going to invent any — the numbers in circulation about agentic commerce volumes are mostly vendor projections, and we treat those the way we treat everything else on our statistics page: with the source and method attached, or not at all.

What we do know is narrower and more useful. Businesses that intended to welcome agents are blocking them by accident. A default that reaches a fifth of the web now turns agents away from any page carrying an ad. And there is no standard way to say yes yet.

The cost of checking is half an hour. The cost of not checking is a category of visitor you never knew arrived and never knew left.

Frequently asked questions

1 Are websites really blocking AI shopping agents?

Yes, and some of it is deliberate. TechCrunch reported on 6 October 2026 that Amazon began blocking Meta's Muse agent from its retail site around 21 September, citing Bloomberg. United points to Terms of Use that bar robots and automated processes without prior written permission. Yelp does not permit non-human traffic unless the agent has paid for access through its data licensing programme. eBay says it restricts unauthorised agents.

2 Can a website block an AI agent without meaning to?

Easily, and this is the more common case. Walmart is a Muse partner and its customers' purchases were still failing. The apparent cause was a human-verification button which, when interrupted, fails the check and turns the agent away. Zillow says it allows agents acting for consumers but that its anti-bot measures can occasionally block some of them anyway. Neither of those is a policy decision. Both are a settings outcome.

3 What did Cloudflare change about AI agents?

Cloudflare's preset for ad-supported domains allows search crawlers, refuses AI training, and blocks user-directed agents on any page showing an ad. The stated reasoning is that an ad indicates the page was built for a person. There is currently no opt-out directive for agents, only a block. The default reaches new domains, new sites from existing customers, and free-tier accounts. Cloudflare fronts roughly a fifth of the web.

4 Is blocking AI agents the same as blocking AI crawlers?

No, and conflating them is the mistake we are seeing most often. A training crawler collects material to build a model. A search crawler indexes pages so they can be surfaced later. A user-directed agent is acting for a specific person who is trying to do something right now. Cloudflare split its single Block AI Bots switch into three separate controls precisely because these are different things with different consequences.

5 What should a business owner actually do about this?

Find out what your CDN or bot-protection product currently does with agent traffic, because you may never have chosen it. Then decide deliberately. For a content or publishing site the answer may reasonably be to keep blocking. For a site where someone might buy something, book something or request a quote, turning away a visitor who arrived with purchase intent is a different trade entirely.

6 Is there a standard coming for this?

There are three, and the oldest is Google's. Google announced the Universal Commerce Protocol on 11 January 2026, co-developed with Shopify, Etsy, Wayfair, Target and Walmart, and it is the only one already shipping to consumers — Universal Cart began rolling out around 15 October 2026 in the Gemini app and through AI Mode. Sierra and Meta announced the Personal Agent Protocol on 6 October 2026, built on OAuth, covering guest and signed-in sessions with read-only or write access, and explicitly excluding payments from its first version. Visa runs a third, the Trusted Agent Protocol, joined by Microsoft, Stripe, Shopify and Worldpay. Stripe appears in all three.

#AI agents #agentic commerce #Cloudflare #bot management #AI search #e-commerce #technical SEO #Universal Commerce Protocol #Google

Take the next step

Ready to Apply These Insights?

If you do not know what your CDN currently does with agent traffic, that is the check worth running this week. A Visibility Diagnostic covers crawler and agent access alongside the rest of your AI search footprint.

Start Your Free Audit →
Book a 15-Minute Strategy Call →