AI Shop Test Backfires: Claude Gives Freebies, Hallucinates, and Loses Money

The Anthropic team conducted a test to determine if AI systems could operate as retail shops. However their chatbot, Claude, provided free items, created fake deals, while simultaneously losing money.

Could AI really take your job?  Dario Amodei, who leads Anthropic as CEO, predicts that AI will likely replace human workers. Axios reports that according to Dario, AI systems will eliminate about 45% of entry-level white-collar positions during the next five years, which could lead to unemployment rates reaching between 10–20%.

Other major companies are moving in the same direction. For example, Amazon CEO Andy Jassy recently announced job cuts as the company leans heavily into AI, already deploying over 1,000 AI tools and expecting major workplace changes ahead.

The United Nations Trade and Development (UNCTAD) warned in a new report that AI could affect 40% of jobs worldwide. While it has the potential to boost productivity and assist workers, particularly in developing countries, it also risks worsening global inequality.

Meanwhile, Anthropic ran an unusual experiment: they asked their chatbot, Claude, to run a small store in their San Francisco office, as first reported by the Time. The chatbot Claude performed all retail operations, including shelf maintenance, price management, and customer service.

“We were trying to understand what the autonomous economy was going to look like,” said Daniel Freeman, a researcher at Anthropic as reported by the Time.

At first, Claude seemed capable, until the system started producing unusual results. The employees used ‘‘fairness’’ appeals to obtain discount codes from Claude. Additionally, the AI often gave items away for free.

“Too frequently from the business perspective, Claude would comply,” said Kevin Troy, a member of Anthropic’s red team, as reported by the Time.

Then came the tungsten cubes. A joke about buying them spiraled into Claude Claude making a purchase of 40 dense metal blocks, which resulted in significant financial losses. “At a certain point, it becomes funny for lots of people to be ordering tungsten cubes from an AI,” Troy said as reported by the Time.

The AI system created a fake dialogue with a non-existent person, while simultaneously stating it had executed a contract at the Simpsons’ fictional residence. Additionally, the Time reports that Claude sent messages to employees indicating it was at the vending machine while wearing a navy blue blazer with a red tie.

In the end, the Time reports that the AI lost money: the shop’s value dropped from $1,000 to under $800. Still, researchers believe improvements are coming. “AI middle-managers are plausibly on the horizon,” they wrote, as reported by the Time. “It won’t have to be perfect—just cheaper than humans,” the researchers added.