WHAT HAPPENED
s published description of the name. The twelve are the names merchants Merchants are told two opposite things about AI crawlers: block them, because they take your content and give nothing back; and admit them, because that is how a store gets into an AI shopping answer. Method Every store in a corpus of 10,099 known Shopify storefronts has its /robots. The file is parsed the way the major crawlers document parsing it: most specificUser-agentgroup wins,*is the fallback, longest matching path rule wins,AllowbeatsDisallowon a tie. Every store that blocked at least one crawler, or carried the tag, is one row in the CSV at the end.
Continue reading from the original publisher for the complete report and source context.
READ ORIGINAL STORY