Professor Kyoungmu Lee, SNU Professor Kyoungmu Lee develops deep learning-based image reconstruction technique (Science 21, 20180509)

2018-05-14l Hit 881

Going beyond the limits of super resolution image restoration
SNU Professor Kyoungmu Lee presents deep learning-based image restoration technique

Several years have passed since sensational deep learning-based technologies, such as AI and AlphaGo, were first introduced to the world, but once again another innovative application of deep learning was developed. Kyoungmu Lee, professor of electrical and computer engineering at SNU, is at the center of this innovation where the possibility of overcoming the limitation of current image restoration techniques with the help of deep learning was proposed.

Successful development of top-notch image reconstruction techniques based on deep learning

Professor of electrical and computer engineering of SNU, Kyoungmu Lee, has recently developed a super-resolution image reconstruction technology based on deep learning to accurately recover images with low-resolution. Deep learning is a technology that allows computers to make use of data and artificial neural network to train itself. The suggested novel image reconstruction technology, which uses deep learning, is highly praised for its contribution to the field of computer vision, which is an analysis system that deals with how to make machines and computers perceive and understand like human visual systems.

There have been consistent demands on this type of deep learning-based super-resolution image reconstruction techniques due to its potential applications in various fields. In this context, Professor Lee’s work carries even greater significance.

Image processing techniques that use computers to handle visual information in the way that human visual system perceives objects and surrounding environments, also known as Computer Vision, is considered as the most important research field in Artificial Intelligence. Recent innovations in deep learning also come from the field of computer vision. Even before deep learning technology drew our attention, Professor Lee has conducted research related to image processing and computer vision for a long time leading to this remarkable result.

Talking about the research background, Professor Lee said, “Working with companies and research institutes gave me the insight that in real-life situations, where actual practical problems are dealt with, the enhancement of image resolution is very important. Realizing that the key solution for this would be improving the image qualities in actual use cases, I focused my interest and research on image reconstruction using recent deep learning techniques.”

Super-resolution image reconstruction uses software techniques to precisely recover the original high-resolution image from low-resolution images suffering from information loss. In forensic videos for example, if face recognition or license plate detection is possible with resolution enhancement of the target object and noise filtering, it can significantly help the investigation process. Situations like this are where Professor Lee’s work can be applied to overcome the existing limitations of image restoration.

Most of the current CCTV cameras are limited to VGA resolution, and even when HD resolution is supported, it is hard to identify distant objects due to the low resolution of the corresponding area. Also, with the increasing demands on higher resolution CT, MRI, X-ray and ultrasound images for accurate detection and diagnosis, and other applications such as analyzing bio images, satellite, and aerial photographs, super-resolution image reconstruction technologies are important without a doubt.

Innovative super resolution image reconstruction algorithm

Immersed in research based on ingenious ideas such as image enhancement algorithms using artificial neural networks, Professor Lee unfolded numerous deep learning-based algorithms, including Very Deep Super Resolution network (VDSR), Deep-Recursive Convolutional Network(DRCN), Enhanced Deep Super-Resolution network(EDSR), and Multi-scale Deep Super-Resolution system (MDSR), bringing substantial performance improvement.

VDSR system is the first to use a very deep convolutional neural network consisting of more than twenty layers with the concept of residual connection. A deeper convolutional neural network can exploit broader contextual information of the input image, which makes it possible to extract more meaningful and useful features for enhanced results. On top of that, using a novel concept called residual connection, where the input values are delivered unchanged and the training is done with only the difference between the output and the input, it accelerates the learning rate and shows reliable performance. Later on, the performance and speed were once again substantially improved through expanding this to ESDR and MDSR with more than 60 layers.

This technique provides software solutions to enhance the resolution of low-quality images. Previous solutions for the image reconstruction problem mainly focused on mathematical attempts to simplify the model. By using network training instead of only relying on mathematical analysis, the limitations of past methods could be solved. This can also be used to diminish the blurring effects caused by subtle camera movements and moving objects, which were hard to tackle with previous image reconstruction techniques.

Professor Lee commented that his research shows the potential that the image reconstruction problem, thought to be difficult until now, can be solved.

Professor Lee explained, “Previous solutions had shown meager performance improvements. Even when complicated mathematical equations were applied, it did not have much effect and only increased the desire for new methods. By developing techniques using deep learning, we could make dramatic performance improvements.” He also added that such techniques can significantly change the cost and effects of current solutions.

Global impact of the super resolution image construction technique

Professor Lee’s work is attracting attention not only from Korea but from worldwide by proving its competency at premier computer vision and machine learning conferences. Through the research on super-resolution image reconstruction issues, his team developed and presented algorithms with unprecedented performance, such as VDSR, DRCS, EDSR, and MDSR, attracting massive interest.

The team delivered oral presentations on two papers at the world’s best computer vision and machine learning conference CVPR 2016, and also proved the significance of their research through winning the “2017 NTIRE challenge on super-resolution”, where 21 global teams competed.
The VDSR paper is also globally considered as a representative paper in the area of super-resolution image reconstruction, which was cited more than 400 times during one and a half years. It has a high impact factor reflecting the fact that it is one of the most cited papers from CVPR 2016.

Numerous potential applications of the image restoration technique

This research is highly useful in the sense that it can be applied to all areas that use images. In particular, the super-resolution image reconstruction technique that Professor Lee developed is expected to be applied in various areas such as transportation, crime prevention, science, military, space, and social media.

For example, it can be used to improve the image quality of surveillance cameras and black boxes to support criminal investigations and traffic analysis. Also, it can allow UHD TVs to provide higher resolution videos, enhance smartphone screen resolution, and improve the accuracy for semiconductor flaw detection. In the medical field, which has a direct impact on humans, this technology can enhance medical images to help life extension. Furthermore, predictions were made that it will be the core technology for autonomous driving, intelligent robots, drones, military, and satellite image analysis.

Professor Lee said, “Super-resolution techniques can be applied to all areas where high-quality images need to be obtained still using low-cost cameras. It can be exploited in many more areas where visual information is utilized.”

World’s best super resolution algorithm based on deep learning

Taking the current research one step further, Professor Lee’s goal is to develop top-notch super-resolution algorithms for actual deployment in real life situations. In addition, he is working towards analyzing how far super-resolution quality improvement can be reached using deep learning with logical reasoning, and also figure out its limitations.

Through this research, he achieved huge success by solving the problems of super-resolution which were considered to be difficult until now. Based on this, knowing the fact that this research can potentially solve various image restoration problems, he is aiming to solve difficult problems such as noise filtering one by one.

Professor Lee said, “Our team is currently trying to tackle the critical issues in computer vision. In terms of image reconstruction, our goal is to find an overall solution for image quality problems, such as blurring and noise. Ultimately, we would like to create an artificial intelligence system that can automatically produce high-quality images from any input images.”

Other than developing algorithms for image restoration, he is also interested in the recent topics related to deep learning, such as system models that can produce authentic images and automatic machine learning algorithms like self-adaptive learning and meta-learning.

The key to success is creativity

Professor Lee revealed that the key to the success of this research was creativity. Research in artificial intelligence asks for a higher level of creativity compared to other areas. Professor Lee himself always considers this as most important during his own research and tries to promote his students to always enjoy their research and to be innovative.

Professor Lee commented, “The focal point of successful research is creativity. Creativity can be unleashed only when you are fully devoted to the problem and have enthusiasm. My students were passionate and conducted active research through open discussions which led to successful research results in a short time.”

According to Professor Lee, the super-resolution image restoration problem, which is in the center of attention worldwide, is just the beginning for overcoming the limits of current image restoration techniques using deep learning.

It is said that this super-resolution image restoration technique is rather a turning point than a conclusion. Also, it is a proven momentum that inspired many of the current researchers. This research made a small yet strong contribution to Professor Lee’s ultimate research goal, “Artificial Intelligence for Humans”, which is to solve the numerous problems the current society is facing.

Written by Kyungjin Lee, Editor of Department of Electrical and Computer Engineering,