Add news
March 2010April 2010May 2010June 2010July 2010
August 2010
September 2010October 2010
November 2010
December 2010
January 2011
February 2011March 2011April 2011May 2011June 2011July 2011August 2011September 2011October 2011November 2011December 2011January 2012February 2012March 2012April 2012May 2012June 2012July 2012August 2012September 2012October 2012November 2012December 2012January 2013February 2013March 2013April 2013May 2013June 2013July 2013August 2013September 2013October 2013November 2013December 2013January 2014February 2014March 2014April 2014May 2014June 2014July 2014August 2014September 2014October 2014November 2014December 2014January 2015February 2015March 2015April 2015May 2015June 2015July 2015August 2015September 2015October 2015November 2015December 2015January 2016February 2016March 2016April 2016May 2016June 2016July 2016August 2016September 2016October 2016November 2016December 2016January 2017February 2017March 2017April 2017May 2017June 2017July 2017August 2017September 2017October 2017November 2017December 2017January 2018February 2018March 2018April 2018May 2018June 2018July 2018August 2018September 2018October 2018November 2018December 2018January 2019February 2019March 2019April 2019May 2019June 2019July 2019August 2019September 2019October 2019November 2019December 2019January 2020February 2020March 2020April 2020May 2020June 2020July 2020August 2020September 2020October 2020November 2020December 2020January 2021February 2021March 2021April 2021May 2021June 2021July 2021August 2021September 2021October 2021November 2021December 2021January 2022February 2022March 2022April 2022May 2022June 2022July 2022August 2022September 2022October 2022November 2022December 2022January 2023February 2023March 2023April 2023May 2023June 2023July 2023August 2023September 2023
News Every Day |

Testing AI or Not: How Well Does an AI Image Detector Do Its Job?

You may recall earlier this year when many social media users were convinced pictures of a “swagged out” Pope Francis—fitted with a white puffer jacket and low-hanging chain worthy of a Hype Williams music video—were real (they were not).

An AI-generated image of Pope Francis, created using MidJourney.

These images were the product of Generative AI, a term that refers to any tool based on a deep-learning software model that can generate text or visual content based on the data it is trained on. Of particular concern for open source researchers are AI-generated images.

DALL-E, Stable Diffusion, and Midjourney—the latter was used to create the fake Francis photos—are just some of the tools that have emerged in recent years, capable of generating images realistic enough to fool human eyes. AI-fuelled disinformation will have direct implications for open source research—a single undiscovered fake image, for example, could compromise an entire investigation.

Earlier this year, the New York Times tested five tools designed to detect these AI-generated images. The tools analyse the data contained within images—sometimes millions of pixels—and search for clues and patterns that can determine their authenticity. The exercise showed positive progress, but also found shortcomings—two tools, for example, thought a fake photo of Elon Musk kissing an android robot was real.

Bellingcat sought to repeat this experiment with open source research in mind. We tested one of these tools- AI of Not –  on 200 images: half of them real, half AI-generated. In particular we wanted to understand how effective the tool was at detecting images with watermarks and images which have been compressed – two specific challenges often faced by open source researchers:. The results show a technology suited to navigating watermarks but that struggles with images that have been compressed.

The Tool

The tool is called AI or Not. Developed by Optic, an American tech company founded by former Google product director Andrey Doronichev, it “uses advanced algorithms and machine learning techniques to analyse images and detect signs of AI generation.” It has been trained on DALL-E, Midjourney, Stable Diffusion, generative adversarial networks and face image generators.

The company states that the tool is designed to provide highly accurate results.

The Method

For the test, Bellingcat fed 100 real images and 100 Midjourney-generated images into AI or Not. The real images consisted of different types of photographs, realistic and abstract paintings, stills from movies and animated films and screenshots from video games. The Midjourney-generated images consisted of photorealistic images, paintings and drawings. Midjourney was programmed to recreate some of the paintings used in the real images dataset. 

The AI image dataset is available here. The real image dataset is available here. What follows are the results.

First Impression

AI or Not was highly accurate during the first round of identifications. 

It was particularly adept at identifying AI-generated images — both photorealistic images and paintings and drawings. 

AI or Not successfully identified visually challenging images as having been created by AI. 

For example, someone without professional art training might struggle with correctly identifying this modernist-style mural as having been generated by Midjourney: 

A modernist-style mural, created by Midjourney.

AI or Not was also successful at identifying more photorealistic Midjourney-generated images, such as this photorealistic aerial image of what is supposed to be a frozen Lake Winnipeg in Manitoba, Canada. 

An image of a frozen Lake Winnipeg, created by Midjourney.

It also successfully identified AI-generated realistic paintings and drawings, such as the below Midjourney recreation of the famous 16th-century painting The Ambassadors by Hans Holbein the Younger.

A recreation of Hans Holbein the Younger’s The Ambassadors, created by Midjourney.

The real painting is exhibited in the National Gallery in London, UK. 

Overall, AI or Not correctly detected all 100 Midjourney-generated images it was originally given. 

The tool also maintained a high detection rate for real images although it also produced false positives: incorrectly identifying real images as having been generated by AI. 

AI or Not successfully identified the following real images:

  • 20 images taken by Bellingcat contributor Dennis Kovtun
  • 20 paintings by various artists from the Renaissance period
  • 19 out of 20 abstract paintings
  • 10 of 10 movie stills
  • 10 out of 10 animation stills
  • 10 out 10 video game screenshots from the late 2000s and early 2010s

AI or Not produced some false positives when given 20 photos produced by photography competition entrants. Out of 20 photos, it mistakenly identified six as having been generated by AI, and it could not make a determination for the seventh. 

All the photographs that AI or Not mistakenly identified as AI-generated were winners or honourable mentions of the 2022 and 2021 Canadian Photos of the Year contest that is run by Canadian Geographic magazine. It was not immediately clear why some of these images were incorrectly identified as AI.

Generally, the photos had a high resolution, were really sharp, had striking bright colours and contained a lot of detail. Several had unusual lighting or a large depth of field, and one was taken using long exposure. 

Based on the above test, we concluded that AI or Not is quite good at identifying real images — it successfully identified real paintings and drawings and most real photographs, though it may struggle with some high-quality photos.

While AI or Not is, at first glance, successful at identifying AI images, there’s a caveat to consider as to its reliability. 

AI or Not: Compressed and Watermarked Images

Every digital image contains millions of pixels, each containing potential clues about the image’s origin. 

But what happens if a chunk of this valuable data is lost or distorted? During the first round of tests on 100 AI images, AI or Not was fed all of these images in their original format (PNG) and size, which ranged between 1.2 and about 2.2 megabytes. When open-source researchers work with images, they often deal with significantly smaller images that are compressed. 

When an image is resized or distorted, or when its resolution is lowered, the pixels in it are altered and the “digital signal” that helps a detector identify the origins of the image is lost, Kevin Guo, the founder of image detection tool Hive, told the New York Times. 

Hive provides deep-learning models for companies that want to use them for content generation and analysis, which include an AI image detector. It also has a free browser extension, but the extension’s utility for open-source work is limited. It was “unable to fetch results” on Telegram, while a small pop-up window showing the probability that an image is AI-generated did not open on X, the social media site formerly known as Twitter. The window did open on Facebook.

To test how well AI or Not can identify compressed AI images, Bellingcat took ten Midjourney images used in the original test, reduced them in size to between 300 and 500 kilobytes and then fed them again into the detector. 

This produced mixed results. AI or Not falsely identified seven of ten images as real, even though it identified them correctly as AI-generated when uncompressed. 

For example, when compressed, this Midjourney-generated photorealistic image of a grain silo appears to be real to the detector.

An image of a grain silo, created by Midjourney.

Out of ten compressed images, seven were photorealistic. AI or Not was particularly unimpressive with them — it falsely identified all seven as the work of a human. It fared better with drawings and paintings: when given three, correctly identified them, even when compressed. 

With real compressed images, AI or Not fared better. It incorrectly identified two out of three paintings as AI-generated. However, it successfully identified six out of seven photographs as having been generated by a human. It could not determine whether an AI or a human-generated the seventh image. 

Bellingcat also tested how well AI or Not fares when an image is distorted but not compressed. In open source research, one of the most common types of image distortions is a watermark on an image. An image downloaded from a Telegram channel, for example, may feature a prominent watermark. 

Bellingcat took ten images from the same 100 AI image dataset, applied prominent watermarks to them, and then fed the modified images to AI or Not. The images were not compressed. 

An image of a pyramid, created by Midjourney (watermark added by Bellingcat using Watermarkly).

AI or Not successfully identified all ten watermarked images as AI-generated. The watermarks did not throw the detector off.

Based on this sample set it appears that image distortions such as watermarks do not significantly impact the ability of AI or Not to detect AI images. Compression, on the other hand, plays a significant role. The larger the image’s file size and the more data the detector can analyse, the higher its accuracy.  

Great or Dud?

AI or Not appeared to work impressively well when given high-quality, large AI images to analyse. Its performance seems to be unaffected by watermarks.  

It maintained a good success rate with real images, with the possible exception of some high-quality photos. It also worked well with compressed real images. 

However, the positive results with real images are tempered by a comparatively unimpressive performance with compressed AI images. This point in particular is relevant to open source researchers, who seldom have access to high-quality, large-size images containing lots of data that would make it easy for the AI detector to make its determination. 

Instead, they often work with significantly compressed images. The fact that AI or Not had a high error rate when it was identifying compressed AI images, particularly photorealistic images, considerably reduces its utility for open-source researchers. While AI or Not is a significant advancement in the area of AI image detection, it’s far from being its pinnacle. 

Bellingcat is a non-profit and the ability to carry out our work is dependent on the kind support of individual donors. If you would like to support our work, you can do so here. You can also subscribe to our Patreon channel here. Subscribe to our Newsletter and follow us on Twitter here and Mastodon here.

The post Testing AI or Not: How Well Does an AI Image Detector Do Its Job? appeared first on bellingcat.


«Вся перекошена»: паралимпийцу дважды сломали инвалидную коляску при перелете

Top 10 Underrated Members Of The X-Men

All The Ways Mt. Everest Can Kill You WIRED

3 Body Language Experts Spy On Dates & Guess How Long Couples Have Been Together WIRED

Natural Selection: Life's Way of Stayin' Alive: Crash Course Biology #13

Read also

One run gets it done: White Sox 1, Red Sox 0

Gemma Winter betrays Jenny Connor over the Rovers in Coronation Street

UFC Fight Night 228 video: Marina Rodriguez TKOs bloody Michelle Waterson-Gomez, calls out Tatiana Suarez

News, articles, comments, with a minute-by-minute update, now on

News Every Day

Natural Selection: Life's Way of Stayin' Alive: Crash Course Biology #13 — latest news 24/7. You can add your news instantly now — here

News Every Day

Gluten-Free Crepes - Easy Refined Sugar-Free Almond Oat Crepes Recipe

Sports today

Новости тенниса

Теннисист Карацев обыграл Маррея и вышел в четвертьфинал турнира ATP в Китае

Спорт в России и мире

«Зенит» пробился в финал Суперкубка лиги ВТБ, обыграв «Нижний Новгород»

All sports news today

Sports in Russia today


Гол Бонгонды принес "Спартаку" победу над "Динамо" в матче девятого тура РПЛ

Новости России

Game News

Defend your interstellar factory when Dyson Sphere Program gets combat this December


Заступил на пост: Собянин продолжит развивать Москву

Губернаторы России
Игорь Маковский

Александр Богомаз и Игорь Маковский в Брянске провели рабочую встречу и посетили героя-энергетика

Наука помогать: в регионах России в рамках акции «Поделись своим знанием» пройдут уроки волонтерства

Самолет из Москвы экстренно сел в Перми из-за ухудшения состояния пассажира

Джонни Депп меняет профессию: как живет российский двойник голливудского актера

Москвич чуть не сжег Mercedes, пытаясь произвести впечатление на друзей

Финалист международного конкурса пианистов и уроженец Читы Иван Чепкин выступит в Забайкалье вместе с заслуженной артисткой РФ Екатериной Мечетиной и Петром Лундстремом

Проект "Мой Рахманинов" будет представлен в Якутске

Певец SHAMAN записал обращение девятилетнему белгородцу

Концертом фортепианной и вокальной музыки Магаданская областная филармония открыла творческий сезон

Александрова не смогла пробиться в 1/4 финала турнира WTA 1000

Теннисист Карацев обыграл Маррея и вышел в четвертьфинал турнира ATP в Китае

Рублев уступил американцу Фритцу в матче Кубка Лейвера

Карацев с победы стартовал на турнире ATP 250 в Чжухае

Переменная облачность и до 25°C тепла ожидаются в Москве 24 сентября

Врачи перинатального центра в Подмосковье научились спасать зрение недоношенных детей

Синоптик Леус: в Москве в субботу ожидается до +25°C

Суперкубок Единой лиги ВТБ. 1/2 финала. 25 очков от Акун-Перселла помогли «Локомотиву-Кубань» обыграть «Фенербахче», «Нижний Новгород» крупно уступил «Зениту»

Гол Бонгонды принес "Спартаку" победу над "Динамо" в матче девятого тура РПЛ

Сергей Собянин рассказал об уникальной разработке московских инженеров

Собянин: свыше 900 московских компаний стали крупным бизнесом в этом году

Игорь Крутой, Лариса Долина, Дима Билан и другие российские звезды на гала-концерте благотворительного детского фестиваля «Добрая волна»

Во Мценске мечтают найти останки динозавров

Байден обвинил республиканцев в угрозе прекращения работы американского правительства

Заступил на пост: Собянин продолжит развивать Москву

Почти 200 нижегородцев не могут вылететь из Египта из-за отмены рейса

Путин в России и мире

Персональные новости
Елена Маркелова

Мастер-класс для ветеранов СВО и членов семей военнослужащих прошел в Самарском филиале Фонда "Защитники Отечества"

News Every Day

Gluten-Free Crepes - Easy Refined Sugar-Free Almond Oat Crepes Recipe

Friends of Today24

Музыкальные новости

Персональные новости