Vuk Radovanovic

My first research project was a ton of fun to work on. The goal was to render into a compressed representation of an original RGB image, rather than rendering the RGB itself.

Visual (image/video) generative models are pretty heavy on compute, so the most successful ones usually don’t operate directly in RGB. They use a model referred to as a Variational Autoencoder or VAE for short, which consists of an Encoder and Decoder. The Encoder can take an RGB image, and output a latent distribution that is compressed in resolution, but expanded in channel count. (e.g. 1024x1024x3, where 3 is RGB channels, gets encoded to 128x128x16, for Stable Diffusion 3.5.) The Decoder can take a latent and then reconstruct the RGB.

Lamp latent An example of the latent space. Green is more +, red is more -, black is 0.

Reaching those latents is the goal! In short, we modified a renderer to support 16 channels, and made some modifications to the rendering equation to better represent all of the common behaviors of the latents as well.

The end result is a system that can take any 3d scene, render it from one view + encode to get a single ground-truth latent, then find ideal scene parameters to generate new latents from new camera angles or with changes made to the scene (light movement, object movement).

You can find more information on the latent rendering project website, read the paper on arXiv, or watch the video below.

Also, our source code is available here!