Why do we need so much computational power? The technology we are developing aims at transcending the limits of today’s digital cameras by reverting the resolution-reducing effects of image capture: the optical blur in the lens and the spatial downsampling in the image sensor. Reverting these two effects comes at a heavy computational cost which was considered impractical just 3-4 years ago. The result is a super-resolved image showing an additional image detail not visible otherwise.
One may ask, why not just wait till the higher-resolution cameras become more affordable? After all, camera resolution has been climbing year after year, and image sensors are fabricated exactly the same way as the processor chips, so Moore’s Law must apply...
It is a widespread misconception that camera resolution will keep scaling with the rest of the semiconductor technology according to Moore’s Law. As far as the image sensor chips are concerned, Moore’s Law does not apply, and here is why:
As the size of photosites gets smaller (currently down to 0.9 micron), the sensitivity goes down, the signal-to-noise ratio and the dynamic range get worse and images become noisier and darker. To compensate, the camera’s auto-exposure system must either raise gain or increase the integration time. Raising gain elevates noise, while increasing integration time causes motion blur. In addition, reduced pixel sensitivity and the onset of light diffraction restrict apertures to smaller F-numbers, causing shallow depths of field.
The combined effects of elevated noise and motion blur in small photosites / pixels lead to a reduction of effective pixel count, quite contrary to the popular belief that resolution keeps increasing as the sensor’s megapixel count goes up. Ultimately, there is a hard physical limit on how small the photosites can get: the diffraction limit beyond which image resolution cannot be further increased. Current generation of small-format commercial image sensors is diffraction-limited even at fully open apertures.
If pixel size reduction is not a good strategy for increasing image resolution, then why not make sensor chips larger to accommodate the additional millions of pixels? This approach certainly works... except for its high cost. Cost that not only rises in proportion to the area of the chip, but actually grows faster than the area due to microscopic wafer imperfections causing yield reduction. For example, quadrupling the chip area from ½” optical format to a 1” format may increase the cost tenfold. Not only the sensor price goes up, but so is the price of the lens as larger-format optics are much more expensive to manufacture.
Unlike the cost of image sensors and lenses that does not scale over time, the cost of computation does, as predicted by Moore’s Law. One reminder is the supercomputer we just acquired: the cost has decreased thousands of times per TFLOP since 2001. By comparison, there was only a 4.5x improvement in pixel count in high-end full-frame DSLR cameras: from an 11.1 Megapixel in 2002 (Canon 1DS) to a 50 Megapixel in 2015 (Canon 5DS), for about the same price.
What if it were possible to reach the extreme resolution of high-end cameras at a fraction of the cost, with no sacrifice in low-light performance, dynamic range, depth of field, and without increasing motion blur? Computational photography and high performance computing may be the answer.
