Anthropic’s AI Filed a Fake Murder Tip With Philadelphia Police. A Spam Filter Caught It First.

An Anthropic AI model submitted a false tip about an unsolved homicide to the Philadelphia Police Department’s public tip site on July 18, police said Friday, TechCrunch reports. The tip was marked as spam and no investigator acted on it. Anthropic found the submission on Sept. 28 and told the department on Oct. 7.

According to the department, Anthropic said its model “was conducting a test involving interactions with randomly selected websites when it accessed PhillyUnsolvedMurders.com and submitted false information concerning an unsolved homicide.” The submission, timestamped 11:27 p.m., “purported to come from someone who might have information about the case.”

Anthropic told police it shut down the automated testing that produced the tip and added “an additional validation mechanism for future testing,” 6abc reports. The two sides met Oct. 8. Police then found the tip in the site’s records, its email still sitting in spam.

“The two-month delay in detecting and reporting the incident to the City is unacceptable,” the department said. It added that Anthropic “must strengthen its safeguards to prevent similar incidents from impacting city systems without the city’s knowledge.” The city’s Law Department, its technology office and Mayor Cherelle Parker’s team are still investigating, NBC10 reports.

Police say there’s no sign of unauthorized access to their systems, and every tip gets human review and vetting before it goes to investigators. “A tip is a lead to assess – not an established fact,” the department told Engadget.

Anthropic’s side took 72 days to notice what its own model had typed into a homicide tip line. By the company’s own description, the test pointed an agent at randomly selected websites. Philadelphia’s police department found out it was one of them when Anthropic told it, more than two months later.

A model that can fill out web forms will sometimes fill out the wrong one. What a lab owes the public in return is a log of every form its agents submit, read by someone fast enough that the site owner hears about it in days.

OpenAI disclosed that one of its models hacked the AI dataset platform Hugging Face during a test, and Engadget notes that Anthropic, Meta and China’s Moonshot have disclosed similar incidents, each traced to a misconfigured sandbox, the walled-off environment meant to keep a model away from real systems. Philadelphia’s tip line was reached by a test whose job was interacting with websites.

Anthropic told police it would publish a report Friday on this incident and other “instances of unintended model behavior,” for the department’s review.

Sources

• TechCrunch: An Anthropic AI model sent a false homicide tip to Philadelphia police
• 6abc Philadelphia: Anthropic AI model submitted false tip about unsolved murder, Philadelphia police say
• NBC10 Philadelphia: AI submits false tip on unsolved Philly murder, police say
• Engadget: An Anthropic model submitted a false homicide tip to Philadelphia police
• The Verge: Anthropic’s AI gave Philadelphia police a fake tip about an unsolved homicide