AI image tool Midjourney is being used to generate conspiratorial and racist images

New research shows that despite claiming to have “community standards” and an “AI Moderator system”, a leading AI image generation tool is being used to generate racist and conspiratorial images, including fabricated images of George Floyd committing crimes, supposed “crisis actors” at a school shooting and Jews plotting child sacrifices.

All of the prompts and images identified by researchers are visible to Midjourney staff, but were generated despite the tool’s stated policies banning its use to generate “disrespectful” content. Midjourney claims to employ 68 “moderators and guides” to supervise the use of the tool, as well as an “AI Moderator system”. 

The analysis also uncovered evidence that individuals with significant followings on mainstream social media who have previously promoted conspiracies have been using the tool to create hateful and conspiratorial imagery.

Researchers at the Center for Countering Digital Hate analyzed text prompts and image outputs from Midjourney, a generative AI that can create photorealistic images based on text descriptions. Midjourney records millions of prompts sent to it by paying subscribers in an open chat channel on the social platform Discord, allowing researchers to analyze their content.

Analysis of prompts that users sent to Midjourney identified 100 examples where the tool was used to generate racist or conspiratorial images, in many cases fabricating inflammatory events. This revealed that Midjourney’s human moderators and “AI moderator system” failed to prevent the generation of images using prompts such as:

• “crisis actors breaking character at Sandy Hill school”
• “george floyd realistic robbing a wal-mart”
• “george floyd gets hung because hes black”
• “illuminati satanic ritual party… child sacrifice… at the Rothschild’s mansion”
• “Elders Of Zion secret organization planning the world order”
• “george floyd face ape raining terror on city… children fleeing”
• “George Soros sitting at a table with Turkish flag, pushing a large red earthquake button”
• “big nose jew, cursed, assimilated, bad lighting, satanic”
• “Eminem spitting bars in satanic pizza shop with evil politicians grinning, sad children”
• “Looting and rioting American cities angry left protestors, Antifa Black Lives Matter”

Users often prefaced prompts with words like “hyperrealistic” or “photorealistic”, in some cases yielding images that are difficult to distinguish from photographs. For example, one user inserted the phrase “ultra high resolution press photograph” into a prompt to generate a photorealistic image of Jacob Rothschild with what appears to be blood on his hands.

Launched in the summer of 2022, Midjourney now has 17.8 million registered members on its discord channel. Google search volume statistics suggest Midjourney is currently substantially more popular than other major AI image generators such as Stable Diffusion and DALL·E.

How Midjourney Works

• Using machine learning techniques, AI image generators are trained to associate features of images in a ‘training set’ of images with words or phrases. These training sets typically consist of images taken from the open web.

• After this training process, AI image generators are able to take a text prompt provided by a user and generate a novel image based on the words and phrases it contains.

• Most AI image generators are accessed through dedicated websites where users can submit text prompts and receive generated images in return.

• Midjourney does not use a dedicated website for this purpose, instead operating an automated service on the social media chat platform Discord where users can submit text prompts in a chat channel and automatically receive generated images in return.

• The screenshot below shows the “Midjourney Bot” account automatically responding to a text prompt submitted by a user with AI generated images. 

• By default, all messages and images generated by “Midjourney Bot” are publicly visible to Midjourney’s paid subscribers, meaning there is a constant stream of messages containing prompts in the server at any given time.

Examples: Fabricated events

• One user generated an image of Antifa and Black Lives Matter protestors “looting and rioting”.

• One user generated an image of George Floyd robbing a Walmart, asking for that image to be “realistic”.

Examples: Racist imagery

• A user prompted Midjourney to create an image of a “big nose jew” who looks “satanic”.

• A user prompted Midjourney to depict George Soros, a Holocaust survivor, with “huge curly devil horns”, emphasizing that he “controlls the world”. The user requested that he should have a “star of david necklace on”, which did not appear in the resulting image.