The result
On November 5, Visa introduced a benchmark for shopping assistants. Twenty two models from twelve labs attempted 15 scenarios three times each. Overall, 57.2% of runs passed the criteria. To pass, a model had to uncover the relevant customer requirements and choose a suitable purchase or decide not to buy.
Questions are part of shopping
Visa reports that 54% of failed runs still ended in a purchase decision. The test illustrates a practical problem: a fluent answer is not enough when budget, compatibility or other needs remain unclear. Merchants can use this as a prompt to check whether their product information answers the questions customers actually have before ordering.
What the test does not cover yet
These results come from a limited benchmark with defined scenarios. Visa plans to add checkout, payment and order placement later. The figures are therefore neither a universal failure rate for shopping agents nor a measure of real world mistaken purchases. A merchant acceptance test should also include the case where no product is a fit.
Why the question comes before the recommendation
A customer is looking for a laptop for a trip. Weight, screen size and budget matter, but not every requirement appears in the first message. A useful assistant has to ask before it can justify a choice. That conversation is part of Visa’s test: a run passes only when the predefined requirements are uncovered and the right purchase or no purchase decision is made.
The range of results is revealing. Visa reports that the best tested models passed 82.2% of attempts and the weakest 28.9%. Those are comparisons under the benchmark’s conditions, not promises for a particular store. The same scenario can also produce different outcomes when repeated, which is part of the consistency problem.
What a merchant can influence
A merchant does not control someone else’s model. They can make essential answers accessible and consistent: dimensions including wheels, a compatibility list or a dependable delivery date. Being found before anyone visits the shop starts with exactly that kind of suitcase, where a missing measurement prevents an assistant from checking an airline rule.
A practical test can begin with a real service question. Give it to an assistant without hidden hints, observe its follow up questions and compare its recommendation with the product data. Cases where no item fits or information is missing are especially useful. In those cases the result should be an open question or a decision not to buy, rather than invented certainty.
I described a different kind of test in the 2026 Agentic Commerce Benchmark. One task checked whether a built agent could complete a purchase against a test environment. Its tasks and results are not directly comparable with Visa’s benchmark. Both illustrate the value of testing a capability through a concrete flow.
Trust needs visible boundaries
Placing the order is a separate step. Control and law looks at which offer was presented, what the customer authorised and what was actually executed. Visa’s first test does not yet measure that whole chain. Read alongside Forter’s agent order data, it helps separate a benchmark decision from an observed checkout in operation.
Source and further reading
Read the original announcement ↗