It is a common trope in crime and science fiction: an investigator retrieves a blurry photo on a computer screen and requests it to be enhanced. Suddenly, the image sharpens, uncovering a crucial clue. It is a convenient narrative device, but it has long been a frustrating fiction. Enlarging an image too much results in visible pixelation, as there is insufficient data. "Naively upscaling an image will lead to blurriness. While there may be more details, they will be inaccurate, " explained Bryan Catanzaro, vice president of applied deep learning research at Nvidia. However, recent advancements in artificial intelligence algorithms have made image enhancement tools more powerful and user-friendly. Nevertheless, there are still limitations to the amount of data that can be retrieved from any given image. Thankfully, as researchers continuously push the boundaries of enhancement algorithms, they are discovering new methods to overcome these limitations. In the past decade, researchers have utilized a type of AI model called a generative adversarial network (GAN) to enhance images, resulting in detailed and impressive-looking pictures. "The images suddenly improved significantly, " said Tomer Michaeli, an electrical engineer at the Technion in Israel. However, Michaeli noticed that GAN-generated images exhibited high levels of distortion, indicating that they were inventing or "hallucinating" inaccurate details. This observation led to a divergence within the field of photo restoration, with one group focusing on aesthetically pleasing images, mostly generated by GANs, and another group prioritizing data accuracy without much emphasis on visual appeal. In 2017, Michaeli and his graduate student Yochai Blau conducted a formal study on this dichotomy. They plotted the performance of various image-enhancement algorithms on a graph of distortion versus perceptual quality, using a well-established measure for perceptual quality that aligns with human subjective judgment. As anticipated, some algorithms excelled in visual quality, while others were highly accurate with low distortion. However, none of them achieved both advantages simultaneously, requiring researchers to choose one or the other. Michaeli and Blau coined this predicament the "perception-distortion trade-off. " Their research prompted other scientists to develop algorithms that could provide optimal image quality for specific levels of distortion, enabling fair comparisons between aesthetically pleasing algorithms and statistically superior ones. Since then, numerous AI researchers have reported on the distortion and perception qualities of their algorithms, referencing the Michaeli and Blau paper. In certain cases, the consequences of the perception-distortion trade-off are non-critical. For example, Nvidia found that high-definition screens were not effectively rendering lower-definition visual content. To address this issue, they released a tool in February that uses deep learning to upscale streaming video. In this scenario, Nvidia's engineers prioritized perceptual quality over accuracy, acknowledging that the algorithm would invent visual details not present in the original video.

"The model is hallucinating. It's all a guess. Most of the time, it's acceptable for a super-resolution model to make incorrect guesses as long as they are consistent, " explained Catanzaro. However, applications in research and medicine demand higher levels of accuracy. While AI technology has made significant strides in imaging, there are occasional unwanted side effects, such as overfitting or introducing fake features. Therefore, it requires careful handling, as highlighted by Junjie Yao, a biomedical engineer at Duke University. Yao co-wrote a paper in which AI tools improved existing methods of measuring blood flow and metabolism in the brain while staying within the accurate realm of the perception-distortion trade-off. One approach to surpassing the data limitations of a single image involves incorporating data from multiple images. However, this method is often complex and challenging. Researchers studying the environment through satellite imagery have made progress in combining different sources of visual data. In 2021, a group of researchers from China and the UK fused data from two types of satellites to gain a better understanding of deforestation in the Congo Basin, one of the world's largest tropical rainforests and significant biodiversity hotspots. They combined data from Landsat satellites, which have been monitoring deforestation for many years, with deep learning techniques to enhance the image resolution from 30 meters to 10 meters. This improved imagery enabled the detection of 11% to 21% more affected areas compared to using Sentinel-2 or Landsat-7/8 images alone. Michaeli proposes another method to surpass or circumvent hard limits on information accessibility. Instead of settling on a single definitive answer for enhancing a low-quality image, models could present multiple interpretations of the original image. For instance, a fuzzy, low-resolution image of a person wearing a grayish shirt could be reconstructed into a higher-resolution image in which the shirt has black and white vertical stripes, horizontal stripes, or checks – all of which are equally plausible. In another example, Michaeli took a low-quality photo of a license plate and subjected it to a leading AI image enhancer, which indicated that a digit looked most likely to be a zero. However, when processed with a different, more open-ended algorithm designed by Michaeli, the digit appeared equally likely to be a zero, one, or eight. This approach helps eliminate improbable alternatives without erroneously concluding that the digit is zero. As various disciplines tackle the perception-distortion trade-off in their own ways, the question of how much information can be extracted from AI-generated imagery and how much trust can be placed in those images remains central. "We should keep in mind that algorithms simply invent details to produce these visually pleasing images, " cautioned Michaeli. While these hallucinations can be mitigated, the omnipotent "enhance" button, capable of solving crimes effortlessly, will remain a dream.