Super-resolution algorithm in Python

Abstract

Super-resolution algorithms is a family of algorithms used to obtain higher resolution image from set of images in lower resolutions. It's important to notice that by using those methods the higher resolution image is more detailed in opposition to resizing and debluring.

It may be applied on many different data, including spying and meteorological satellites, telescopes, tomographs and X-ray machines. Super-resolution algorithms can be also applied to 3D sceneries or sounds.

In this article, I will describe a super resolution algorithm proposed by Irani and Peleg and its simple implementation in Python.

One of low resolution images (for easier comparison it's resized to fit the estimated image):

low resolution image, super resolution algorithm in Python

The image estimated by using super resolution algorithm:

image reconstructed by using super resolution algorithm in Python

Introduction

The most important role in presented super resolution algorithm plays imaging process, so I will start from its explanation. Let's assume following notation:

Image process of capturing n-th image can be described, by using below formula.

LR(n) = ((T(HR) + N)*hpsf) s↓

It means that any LR image is captured by taking HR image, 1) moving it by T vector, 2) blurring it by noises and hpsf and 3) downsizing it. Multiple different LR images may be obtained from one HR image because T and N may vary.

The goal is, to take this equitation, known factors and low resolution images (its left side) and try to extract HR from it. This may be achieved by below algorithm.

Input: Algorithm:
  • Create initial upsampled image. Simple resizing algorithm with optional debluring may be also used in this step. Let's name this image E.
  • For all LR images simulate image process from E; Different images are taken with different offsets, so simulated LR images will differ.
  • For each LR image get the difference between simulated and captured image, backproject this difference to E.
  • if not extended number of iteration, then back to 2.
Output:
  • E is the SR-image

It's useful to have a function that tells how much estimated image E differs from the original HR image in each iteration. For this purpose we add up for all LR images the differences between values of simulated pixels and captured pixels. This factor is normalized by product of numbers of pictures, width of LR image and height of LR image. This information may be used to tweak the input constants or to compare, how algorithm behaves on different sets of input images.

Software

Python was used due to its easy syntax and support for functional programming. Detailed info, how to use this software was presented on project's wiki page

For easier implementation noises weren't included and psf was hardcoded. More details of motion estimation algorithm used in this project can be found on a separate article.

The super-resolution is available on GitHub

Results

error in super resolution algorithm in each iteration

Conclusion

There are plenty of articles about super resolution algorithms, but almost none of them includes implementation in one of popular programming language. In presented here simple implementation that works correctly and can be used as a base for further researches.

6 comments:

  1. Hi Robert, great post!
    I think I found a typo, though. Your file works great for black and white images, but greyscales all color images. I think the 9 occurances of '(rh, rh, rh)' in super_resolution.py need to be changed to '(rh, gh, bh)' to keep color. I'm not sure if this breaks the algorithm or not, though!

    ReplyDelete
  2. Hi,

    It's intended behavior, current implementation works on monochromatic images, because I was afraid that it will became more error-prone otherwise. I was thinking about that recently and it seams, my afraid doesn't make here. In free time I will update this algorithm.

    Thanks for pointing me to this issue and for nice words about this project!

    regards,
    Robert

    ReplyDelete
  3. Really nice explanation of the basics and has really clarified these algorithms for me. Thanks for the post and the code.

    ReplyDelete
  4. @quetzal_7 and RobertG - perhaps try processing the individual color channels as monochrome, then merging the results for color images?

    ReplyDelete
  5. This code is not in visual basic. Please fix this bug.

    ReplyDelete
  6. @quetzal_7,
    @Peter Dolkens,
    @Anonymous,

    done.

    ReplyDelete