NNSuperResolution is a x4 resolution upscale plugin for Foundry’s Nuke. Enjoy being able to, directly in your composite, upscale photographed/plate material to a much higher resolution. Here is an interactive before/after example:
- High dynamic range upscaling
- Native overscan handling (i.e. it handles larger bounding boxes than the image frame/format)
- Processes the RGB channels of an input image (no support for the Alpha channel yet)
- GPU accelerated using CUDA by NVIDIA (requires a NVIDIA graphics card)
- Internal stitching of several inference patches (to be able to upscale high resolution images without having that much VRAM available on the GPU)
The algorithm that does the upscaling is based on modern neural network technology (also commonly referred to as deep learning, machine learning or artificial intelligence). The models used for this product are trained on stills, which is commonly referred to as single image super resolution. While that is the case, the plugin sometimes produces pretty nice and relatively stable outputs on image sequences as well. That totally depends on the nature of the images that are processed. Please try out the plugin for yourself by visiting the downloads page before you buy a license.
The algorithm that calculates the high resolution output using the lower resolution input image is pre-trained on lots and lots of images during the creation of the software. The only input to the algorithm in this case is the low resolution image, and hence there are no knobs to tweak the upscaled result. The knobs that do exist on the plugin to tweak by the artist are all related to being able to scale up pretty large resolution images using limited VRAM on the graphics card. The neural network requires pretty large amounts of memory even for small resolution input images. To be able to scale up, for example, a 1K input to 4K the image needs to be split up into several passes/patches to fit into memory. This is all done and handled transparently “under the hood” so the artist can focus on more important things. You might need to tweak the settings though depending on the use case and available hardware on your workstation.
Max size sets the maximum dimension in one direction that a patch can have. The default is 500, which means that the max dimension an input patch would be allowed to be is 500×500 px. From our experience that will use up to around 7Gb of VRAM on your graphics card. If you haven’t got that available the processing will error out with a CUDA memory error. To remedy this, and also to be able to input resolutions much higher than a single patch size, you can tweak the max size knob to adapt to your situation. You can lower it to adapt to having much less VRAM available. The plugin will split the input image into lots of smaller patches and stitch them together in the background. This will of course be slower, but it will make it possible to still run and produce much larger results. There is a text status knob above the “max size” knob (in between the two dividers), that will let you know how many inference runs (i.e. how many passes/patches) the plugin will run to create the final upscaled image.
Since the plugin will run multiple patches through the neural network, there will be lots of edges of images present. The edges doesn’t get as satisfying results as a bit further into an image from the edge. Because of this, all these multiple patches that get processed are done so with some spatial overlap. The overlap knob value sets the number of pixels (in the source image resolution space) of the patches’ overlap. The default of 4, which in the resulting high res image is 16 pixels, is usually a good value.
The padding is very connected to the overlap above. While the overlap sets the total amount of pixels the patches overlap, the padding then reduces the actual fading area where the patches are blended to not use the very edge pixels at all. The padding is also specified in pixels in the source resolution, so the default value of 1 actually means that the 4 edge pixels in the high res result of each patch will be thrown away. This way of blending all patches together has proved very successful in our testing.
The last knob is the “multisample” knob. The default is “off” which means that each image patch is run through the upscale algorithm once. This usually produces nice and sharp images. Sometimes it’s more beneficial to get a bit less sharp but smoother result instead. When multisample is “on” it will instead run each image patch through the upscale algorithm 4 times but first pre-process the patch by orienting it differently (using rotations and mirrors). This will of course make the process much slower, but it will in some situations be worth it. You have an example result with multisample on in the before & after image above (the rightmost image) picturing the wetsuit.
Frequently Asked Questions
We have gathered the most common questions about the plugin on a separate Frequently Asked Questions page.
All the downloads are available on the dedicated downloads page.