OpenAI builds abuse classifier with NCMEC – free, but only for approved organizations

The company is releasing a classifier for child sexual abuse imagery, with a dedicated flag for AI-generated content, made free to approved organizations.

File photo: File photo: OpenAI representatives Anna Makanju, Sandro Gianella and Jade Leung meeting Věra Jourová at the European Commission in 2023.
Filbilde: OpenAI-representanter under besøk hos Europakommisjonen i 2023. Selskapet lanserer nå en klassifiserer for seksuelle overgrep mot barn, utviklet med NCMEC. File photo: European Commission - Photographer: Aurore Martignoni, CC BY 4.0 · source

OpenAI builds abuse classifier with NCMEC – free, but only for approved organizations

The company is releasing a classifier for child sexual abuse imagery, with a dedicated flag for AI-generated content, made free to approved organizations. The backdrop is a reporting surge: 400,000 generative AI-related messages to NCMEC's tipline in 2025, up from 67,000 the year before.

What was announced

OpenAI announced at a press briefing in Washington, DC on Monday, November 16, 2026 a classifier the company says can better detect sexual abuse imagery of children – including whether the material is AI-generated. The company is also offering free access for approved organizations, more guidance for developers on how to detect this type of content, and help training law enforcement on how AI can assist investigations (The Verge).

The classifier was developed with help from the National Center for Missing and Exploited Children (NCMEC), the Tech Coalition, and the ROOST project (Robust Online Safety Tools), according to The Hill. The tool is to be combined with manual review.

The problem the tool is meant to address

NCMEC's CyberTipline received 400,000 reports related to generative AI in 2025, an exponential increase from 67,000 in 2024, according to figures NCMEC has provided and The Verge cites. The volume growth is the core of the problem: the demand threatens to overload the tipline's capacity to sort and prioritize the reports.

North Carolina Attorney General Jeff Jackson called the inflow of AI-generated abuse reports to NCMEC "explosive" at OpenAI's event – and warned that if the trend continues, "it will mean that we are failing at our mission" (The Verge).

What OpenAI claims the tool does

OpenAI says the classifier sharply reduces false positives: it flags 48 percent fewer images, while the images it identifies are twice as likely to contain abuse material. These are the company's own figures, not independently verified results.

John Buckley, OpenAI's head of child safety public policy, described the tool as the first broadly available CSAM classifier with a dedicated flag for AI-generated material. The triage value, he explained, lies in distinguishing fictional from real children: "It's important for the industry to be able to understand which images might contain a fictional child rather than a real child" (The Verge).

Access and distribution

OpenAI is deliberately restricting who gets the tool. Buckley told reporters on Monday that the tool must get into the hands of organizations able to handle CSAM, and that OpenAI will not give it to individual developers who lack the required specialized review, or who are not registered to report to NCMEC or other international hotlines (The Hill).

In addition to the classifier, OpenAI is introducing new measures in ChatGPT itself: attempts related to CSAM trigger a new warning, the flagged conversation is removed, and access to ChatGPT is paused for 24 hours. Users retain access to specialist support intended to help them stop harmful behavior. Repeated violations may, according to OpenAI's blog post, lead to account closure or a report to the authorities (as reported by The Hill).

Third-party results – with caveats

Two evaluations are cited, both via OpenAI and participating organizations: the AI platform Fal reports that reviewers confirmed 72 percent of high-confidence flags as previously unidentified CSAM, while SafetyKit reports the classifier detected 90 percent of cases involving drawings or animations that sexualize children, in its evaluation (Fox News).

Fox News notes that the results come from OpenAI and participating organizations, and that independent testing will be needed to determine how reliable the technology is across platforms. It is also unclear whether Fal and SafetyKit used the same datasets or methodology, so the two figures cannot be compared directly.

How it fits together

The combination is logical: as generative models make it easier to produce abuse material, the tiplines are flooded with reports where a large share turn out not to involve real children – material that requires different prioritization than material showing real children. A flag for AI-generated content lets reviewers sort faster. The ChatGPT measure itself – warning, removal, and a 24-hour suspension – is, by contrast, a prevention and deterrence measure, not a detection tool.

What remains unresolved

The effectiveness of the warning and suspension measure is unproven. Fox News writes that it will require continued evaluation. The classifier's performance also rests, for now, on the company's and partner organizations' own figures; no independent audit is cited in the available sources. Nor has technical detail been provided on how the classifier distinguishes AI-generated from real material; according to The Hill's reporting, the tool is to be combined with manual review.

Open questions: how quickly approved organizations will actually get access, how the tool performs on languages and platforms outside those evaluated, and whether the access restriction to registered organizations creates bottlenecks for smaller actors who want to use it.

AIMag.no
AIMag.no
The AIMag.no editorial team covers artificial intelligence, tools, research, and regulation.

Get the best of AI MAG in your inbox

News, analysis, and ideas at the intersection of AI and society.