Skip to content
AI

OpenAI Releases Free CSAM Classifier Built With NCMEC, Adds 24-Hour ChatGPT Suspensions

In early partner tests the tool flagged 48% fewer images but found the same number of abusive ones, OpenAI said.

·2 min read

Prefer Nakoda News on Google

OpenAI has released a classifier, developed with NCMEC, for detecting child sexual abuse images.
OpenAI has released a classifier, developed with NCMEC, for detecting child sexual abuse images.Photo: TechCrunch

OpenAI has released an image classifier that detects child sexual abuse material, including AI-generated images, and is giving select companies free access through its API, TechCrunch reported. The company built the tool with the National Center for Missing and Exploited Children (NCMEC). It also said ChatGPT will now warn users who try to create or access content that sexualises children and suspend their accounts for 24 hours.

The classifier examines previously unseen images for visual signs of abuse and gives each one a score to guide human reviewers on how to handle it. OpenAI is working with the Tech Coalition and ROOST to share access to the tool.

For trust-and-safety teams, the main claim is a lighter workload. OpenAI said that in early testing with partners the classifier flagged 48% fewer images while finding the same number of abusive ones. Images it flagged were twice as likely to be abusive.

In ChatGPT, users who attempt such prompts will see an alert, be directed to specialised support to stop the harmful behaviour and have their account suspended for 24 hours. Repeat offences can lead to permanent removal. ChatGPT already blocks CSAM and requests related to it. OpenAI said it shaped the messages and guidelines around this behaviour with organisations and researchers including Protect Children, the Lucy Faithfull Foundation and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of deterrent messages.

OpenAI is also publishing guidelines for developers who build on its tools, covering how to prevent, detect and help users report CSAM. In early testing, the company said, the guidelines helped a number of developers adopt child-safety tools and register to report material to NCMEC. Law enforcement agencies working on child safety will get AI features including dedicated plugins, investigative tools and complex data analysis, with access managed by programme administrators. OpenAI is also providing training for investigators and a field guide on using AI.

The tools arrive amid growing concern about the safety of AI products. OpenAI faces several lawsuits that blame its technology for harming users, especially children. The safety nonprofit Common Sense Media has labelled OpenAI's technology an "unacceptable risk" for teens. Other chatbots, Grok in particular, have been used to create nonconsensual sexualised images, including of children.

Over the past 12 months OpenAI has released a safety blueprint, updated its content guidelines to bar inappropriate material and retuned its models to avoid producing harmful content. It has also launched a special version of ChatGPT for teens.

By

Nakoda News AI & Tech Desk

AI models, chips, products and research, and what they can and cannot do.

About the desk

Related coverage