Assessing the quality of a single asset

Above: two different lossly compressed versions of an original picture (a frame from Big Buck Bunny). Definition of a criterion on what is an acceptable quality level is not an easy task, hence the plethora of objective quality methods attempting to measure quality.

SSIM time evolution for the first 1240 frames (50 seconds) of Big Buck Bunny, encoded at 500 kbps (above) and 250 kbps (below)

Time evolution of the ratio between SSIMs encoded at 500 kbps and 250 kbps. First 1240 frames (50 seconds) of Big Buck Bunny

SSIM time evolution for the first 1240 frames (50 seconds) of Big Buck Bunny, encoded to 500 kbps with a watermark

Time evolution of the ratio between SSIMs encoded at 500 kbps and with a watermark. In some segments (ratios below 1), watermarks seem to give even better SSIM results than the ‘bona fide’ rendition at 500 kbps.

Bringing in more metrics

Time evolution of the ratio between SSIMs and PSNRs for encodings at 500 kbps and 250 kbps.

Time evolution of the ratio between SSIMs and PSNRs, encoded at 500 kbps and at 500 kbps with a watermark.

Generalizing to several assets

Pairs plot for our dataset of 140 Youtube clips. Distance metrics (cosine, euclidean and Hamming) are fairly linearly correlated, whereas SSIM and PSNR show some logarithmic / exponential relationship with each other. VMAF and MS-SSIM also have some degree of linear correlation to each other. MS-SSIM appears as the one with the most compact distribution.

Spearman correlation table for all seven metrics used in our analysis

Frame from one of the outliers. The video presents a high amount of motion and high detail hard to deal with for the codec in a reasonable compression rate.

Conclusions and further work

References

Livepeer
Livepeer has 30 repositories available. Follow their code on GitHub.
github.com
epiclabs-io/YT8M
(Machine) Learning from YouTube8M dataset . Contribute to epiclabs-io/YT8M development by creating an account on…
github.com
Toward A Practical Perceptual Video Quality Metric
measuring video quality accurately at scale
medium.com
Video Quality Assessment
In Eyevinn’s initiative to share our knowledge around quality we continue with addressing video quality assessment…
medium.com
Human visual system model — Wikipedia
needs additional citations for verification .improve this article by adding citations to reliable sources. Unsourced…
en.wikipedia.org

About the authors