I generated a new night scene from a supplied visual reference, then gave it depth and camera movement. Deep greens, a blue night sky and warm shop lights carry the atmosphere into a different place.
Creative development · AI · Interactive graphics
The camera moves past the foreground tree and planter, revealing the distance to the shop and stairs. Download the generated painting.
Two more places after dark
I generated two new paintings using the original as a visual reference: a tea shop beside a canal and a tram stop on a wooded hillside. Each went through the same depth estimation and inpainting process, with its own mesh and a gently looping camera.
A different kind of light
These next two paintings use different visual languages: bright, opaque storybook color and delicate ink with watercolor washes. I composed both with large foreground shapes, an open middle distance and a clear subject farther away, so a small camera movement has more depth to reveal.
Step into the picture
Choose a scene and open the viewer, then drag to explore or scroll to move closer. The play button returns to the cinematic camera loop. Fullscreen gives the painting more room.
Keeping the character of the painting
The challenge was to add movement without losing the artwork. A fully modeled reconstruction changed the linework and lighting. Generated variations moved away from the original palette. Placing a few painted cutouts at different depths preserved more of the image, but camera movement exposed the gaps and misaligned edges.
Every painting shown here is newly generated using supplied images as visual references. Depth reconstruction keeps each generated painting's visible colors in the scene. The model fills in areas hidden behind foreground objects, so a camera shift can reveal something beyond the edge of a tree.
From illustration to an interactive scene
I adapted the released implementation of 3D Photography using Context-aware Layered Depth Inpainting for this study. Its layered depth representation stores image content at different distances and connects it into a mesh. Learned inpainting supplies color and depth in occluded regions.
The workflow uses the implementation's original MiDaS depth estimation, followed by its edge, depth and color inpainting networks. I first checked the pipeline with the authors' sample image, then processed the alley and compared several camera paths. The cinematic loop uses a much smaller range than those tests.
I exported the resulting mesh for Three.js with the painted colors and no additional scene lighting. The interactive night scenes add a barely perceptible glimmer around the warm shop lights. The viewer pauses offscreen and respects reduced-motion preferences. Every film and the interactive viewer have their own pause and fullscreen controls.
A little movement goes further
This is a 3D photo with a limited viewing range. The single image does not contain every surface of the alley. Move too far and the inferred geometry starts to stretch, particularly around fine foliage. I kept the camera inside a small range so the composition remains the focus.
For the ink drawing, I kept the foreground flowers below the cloud bank so the sky's fine hatching stays unobstructed. Direction and distance are checked against each reconstruction; the camera range is a presentation choice rather than a universal limit of the method.
All five scenes preserve their source paintings' full resolution of 1086 × 1448 pixels. The films are about 8 to 18 MB each. The interactive meshes are about 57 to 63 MB and load only on request. Switching scenes releases the previous mesh. The showcase plays the film; the case study also lets people explore each scene interactively.
Credits
Creative development, research implementation and web viewer: Leo Hoesl.
The reconstruction method is by Meng-Li Shih, Shih-Yang Su, Johannes Kopf and Jia-Bin Huang, presented at CVPR 2020. Research and results · Released implementation.
All five scenes were newly generated with OpenAI's image generation tool using supplied illustrations as style references. The supplied images are not used directly as gallery artwork. My work covers the scene direction, depth reconstruction and interactive presentation.