Visual Quality Intelligence, Part 2: When the Reference Stops Being Gold

Part 2 of a series on how AI learns to see quality. Read Part 1 here.

For decades, pristine original photos were the reference for evaluating image processing and restoration software. Image quality metrics assumed that distance from the reference (distortion) was a good measure of visual quality.

SRGAN broke this assumption in 2017, when its sharper images ranked high in human preference while quality metrics like PSNR preferred blurrier images that looked more similar to the original. SRGAN confronted the field with strong evidence that the reference was no longer a good measure of quality.

An interpretation of this finding was that existing reference-based quality metrics were not aligned with how the human visual system (HVS) evaluates quality. New advances in reference-based quality metrics followed, including development of neural network-based metrics (LPIPS, DISTS, etc.), which proved better aligned with human perception.

However, in 2018, Blau and Michaeli proved that all restoration methods are subject to a universal perception-distortion (PD) tradeoff, regardless of the chosen distortion measure.

The PD tradeoff defines a Pareto frontier for all image restoration methods. When a method is far from the frontier, it is possible to simultaneously reduce distortion and improve perceptual quality (how natural an image looks). For methods near the frontier, improving perceptual quality requires more distortion, while minimizing distortion requires sacrificing perceptual quality.

Does the tradeoff make metrics like LPIPS and DISTS pointless? No. The tradeoff exists for every measure, but its severity varies: semantic measures buy more perceptual quality per unit of distortion than PSNR or SSIM.

The PD tradeoff forces a design choice. Some applications prioritize minimal distortions (e.g., medical imaging); others prioritize perceptual quality (photographic denoising, deblurring). At the frontier, one has to be traded for the other.

For example, a GAN-based image restoration system can define the generator loss as:

l_gen = l_distortion + λ · l_adv

where l_distortion measures difference from the original image, and l_adv, the adversarial GAN loss, measures how well the discriminator can tell the restored image from a real image. In GANs, l_adv is a proxy for perceptual quality. The λ weight is a knob that can be tuned based on the application requirements: λ should be higher for applications that prioritize perceptual quality over low distortion.

This approach makes the PD tradeoff explicit and intentional, and is widely used in state-of-the-art reconstruction models like variational autoencoders (VAEs) used in image and video generation pipelines.

SRGAN's sharper images were a case where distortion worsened while perceptual quality and human preference improved. But increased perceptual quality (how natural an image looks) does not always lead to higher human preference. A blurry but semantically accurate reconstruction of a scene portraying a dog in a park is preferable to a high-quality image portraying a boat on a lake. While the boat scene has higher perceptual quality, it also exhibits extremely high distortion compared to the original dog scene, and would not be deemed an adequate restoration.

Human preference is the ultimate goal in restoration. In practice, neither distortion nor perceptual quality alone can predict human preference, so recent image quality metrics started learning human preference directly from data. The Generalized IQA line of research models human preference as an adaptively weighted combination of distortion (fidelity) and perceptual quality (naturalness).

D(x,y) = F(x, y) + λ(x,y)N(y)

where F is fidelity to the reference, N is naturalness, and the weight λ now adapts to the images themselves. The reference is no longer assumed to be ideal, and relaxing this assumption leads to the A-FINE metric, which aligns better with human preference even when the reconstructed image is better than the reference.

Human preference is the gold standard. Every metric is an imperfect proxy for that standard, with assumptions and blind spots that should continuously be checked against real human opinion.

Next week, Part 3: these imperfect proxies aren't just in papers. One of them quietly decided what your favorite image generator finds beautiful.

How do you judge output quality for your projects: a metric, your eyes, or a vibe check? Leave a comment below or find me on LinkedIn.

References

1. C. Ledig et al., "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network" (SRGAN), CVPR 2017.
2. Y. Blau and T. Michaeli, "The Perception-Distortion Tradeoff," CVPR 2018.
3. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" (LPIPS), CVPR 2018.
4. K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, "Image Quality Assessment: Unifying Structure and Texture Similarity" (DISTS), IEEE TPAMI 2020.
5. D. Chen, T. Wu, K. Ma, and L. Zhang, "Toward Generalized Image Quality Assessment: Relaxing the Perfect Reference Quality Assumption" (A-FINE), CVPR 2025.

This is Part 2 of the Visual Quality Intelligence series. Follow along on LinkedIn or subscribe for the full bibliography with each part.