DeepShop

MolmoWeb 8B 42.3% leads since Mar 2026 Fading

Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement.

Queries are made harder by LLM-driven complexity evolution. Ai2 runs it on Amazon for MolmoWeb.

In the authors’ words

Web agents for online shopping have shown great promise in automating user interactions across e-commerce platforms. Benchmarks for assessing such agents do not reflect the complexity of real-world shopping scenarios, as they often consist of overly simple queries with deterministic paths, such as "Find iPhone 15." Real shopping scenarios are inherently more layered, involving multi-dimensional product attributes, search filters, and user-specific sorting preferences. To address this gap, we introduce DeepShop, a benchmark designed to evaluate web agents in complex and realistic online shopping environments. DeepShop comprises three key components. (1) Query diversity evolution: Starting from real user queries, we generate diverse queries across five popular online shopping domains. (2) Query complexity evolution: We further evolve these queries to increase complexity, considering product attributes, search filters, and sorting preferences, and classify them into three levels: easy, medium, and hard, based on the number of evolutions. (3) Fine-grained and holistic evaluation: We propose an automated evaluation framework that assesses agent performance in terms of fine-grained aspects (product attributes, search filters, and sorting preferences) and reports the overall success rate through holistic evaluation. We conduct a systematic evaluation of retrieval-augmented generation (RAG) methods, web agents, and deep research systems. Results show that RAG struggles with complex queries due to its lack of web interaction, while other methods face significant challenges with filters and sorting preferences, leading to low overall success rates. We also perform cross-category, complexity-based evaluations and error analyses to support the advancement of deep research shopping agents.

Abstract of DeepShop: A Benchmark for Deep Research Shopping Agents (arXiv:2506.02839, CC0)

Made by Lyu et al. (University of Amsterdam and collaborators) Released Jun 2025 Web browsing All tasks public Last in a launch post Mar 2026 , dropped before it saturated Last model added Apr 2026

Level: Everyday Most adults, without training
Scored: Graded by a model A language model scores each response, usually against a rubric.
Answers: Agent task Many steps in a terminal, browser, desktop, app, game or robot.

Paper

Models scored in lab cards

Launch posts list it, from 2 labs

System cards report it, from 2 labs

MolmoWeb 8B leads with 42.3%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

1 MolmoWeb 8B 42.3%
2 Fara-7B 26.2%

#	Model	Score	Setting	Date	Source
1	MolmoWeb 8B	42.3%		Mar 2026	Model card
2	Fara-7B	26.2%		Nov 2025	Model card

2 models have held number one since its release in Jun 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Microsoft Ai2

Lab claims Not tested since Apr 2026

MolmoWeb 8B 42.3%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far

Labs headline it less often now, last in Mar 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Microsoft

Ai2

Date	Launch post	Score	As the post writes it
24 Mar 2026	MolmoWeb 8B	42.3	DeepShop
24 Nov 2025	Fara-7B	26.2	DeepShop

2 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
9 Apr 2026	MolmoWeb 8B technical report	42.3	DeepShop, GPT-4o judge, 100 steps (Table 4)	Capabilities
24 Nov 2025	Fara-7B technical report	26.2	DeepShop success rate, mean of 3 runs, 26.2 ± 2.0 (Table 9, Table 19)	Capabilities

MolmoWeb 8B leads with 42.3%

2 models have held number one since its release in Jun 2025

Scores over time

Labs headline it less often now, last in Mar 2026

2 system cards report it, from 2 labs

More in Computer and browser use