With generative AI, users can create realistic images or alter existing ones in seconds. But the same technology can be misused to produce exploitative or non-consensual content, including material involving children. AI platforms have responded with safeguards designed to block or flag suspected misuse, and in some cases, detected activity can lead to reports that reach law enforcement.
For more than 20 years, The Law Office of Christopher St. John has defended clients facing criminal charges in New Jersey, including AI child pornography cases. As these cases emerge, how an AI platform identifies and responds to suspected misuse can become relevant to a criminal investigation.
How Can Generative AI Be Misused?
AI tools can be misused to produce child sexual abuse material or alter photos of real minors to make them sexually explicit. They can also be used to create non-consensual intimate images or explicit deepfakes of identifiable people.
AI-generated material can also be used in sextortion schemes to threaten or pressure victims, while existing abusive material can be manipulated to create new content. The National Center for Missing & Exploited Children (NCMEC) has also reported AI-related child exploitation involving chat-based activity.
Criminal charges depend on what the material depicts and how it was allegedly created, possessed, or distributed.
What Are AI Developers Doing to Prevent Misuse?
AI platforms use safeguards that can block prohibited prompts or content and flag suspected misuse.
For example, Microsoft says its AI products use filters and classifiers to prevent the generation of child sexual abuse material. Attempts to circumvent those safeguards to generate or share illicit content may result in account suspension and reporting to the National Center for Missing & Exploited Children (NCMEC). Stability AI similarly uses content filters and detection tools and reports detected CSAM to NCMEC through its CyberTipline.
That reporting can extend beyond the platform. NCMEC reviews CyberTipline reports and forwards them to the appropriate law enforcement agency for investigation. As a result, activity flagged by an AI platform can become relevant to a criminal investigation.
AI Safeguards Don’t Determine What Is Legal
Safety controls can restrict what an AI platform allows users to create, but they don’t determine whether conduct is legal. Content that passes through a platform’s filters can still violate the law depending on what it depicts and how it is used.
The opposite is also true. Being flagged or blocked by an AI platform doesn’t necessarily mean the activity was criminal. In New Jersey, criminal liability depends on the content and conduct involved, not simply on how the platform responded.
Talk to The Law Office of Christopher St. John About Your Case
If an investigation involves AI-generated content, one of the first questions is what prosecutors claim the material shows and how they connect it to you. The law that applies will depend on those facts.
The Law Office of Christopher St. John can examine the allegations and evidence involved in your case and explain your legal options. Tell us what happened. Call 856-517-2958 to schedule a free consultation.
Christopher St. John
Tess Berkowitz