How we’re building a Custom 4k Image Upscale System With Hardware Constraints
A look into how we upscale low-resolution images into crisp 4k images
Introduction
One highly requested feature from our users was a 4k image export option for the renders outputted from our Sketch to Render (S2R) tool. This blog will provide a brief overview of how we are designing a system to upscale our S2R image into 4k — the technology, and the constraints.
The Problem
How do we artificially upscale a low-resolution image to 4k while improving the quality? The issue is, the input images are not natively 4k. So how do you take a low-res image and magically turn it into a high-res image?
The Solution
We try to utilize deep learning techniques whenever possible (if it makes sense). Image super-resolution (ISR). ISR techniques refer to the task of enhancing the resolution of an image from low-resolution to high-resolution.
There are many methods used to solve ISR; We chose Super-Resolution Generative Adversarial Network, or SRGAN, which is a Generative Adversarial Network (GANs) that can generate super-resolution images from low-resolution(LR) images, with finer details and higher quality. SRGAN is the first framework capable of inferring photo-realistic natural images for 4× upscaling factors.
You can think of it as what you see in films and series like CSI where someone zooms into an image and it improves in quality and the details just appear. ENHANCE!
Hardware Constraints
A bottleneck we are facing currently is our hardware constraint. We are using a single RTX 6000 to train our models. With this constraint, it is not practical for us to train using the massive datasets necessary to come up with universally generic models— so we have to get creative with our datasets to train specialized models.
Unique data advantage
Vizcom’s S2R tool has various shaders that add unique values to sketches. Each shader has a specific output style that we can predict. Since we know the type of image that will need to be upscaled, we can create these low resolution and high-resolution image pairs ourselves to train SRGAN. This gets rid of any data constraints we might’ve faced otherwise to get the type of results we need for our particular use case.
By training with these specialized datasets, the model will become an expert in upscaling a specific style of image, a style which we can control.
Results
It took plenty of trial and error. Because of our hardware constraint, we have to wait for at least 24 hours+ to see the results of each experiment. After various experiments, we landed on a model that successfully upscaled our renders to 4k while drastically improving the quality. We’re constantly striving to make Vizcom smarter and more effective when improving the quality of our user's creative work.
