Can You Re‑Light a Video After Shooting? JointLight Uses AI for Precise Control of Subject Lighting.

Can you re‑light a video after shooting? JointLight lets AI precisely control character lighting and shadows

World models aim to enable machines to understand how environments change and predict the outcomes of interventions, supporting agent learning, simulation, and content creation. This requires video models to maintain consistent physical relationships when scenes change. Lighting is a key component: when the light source moves, characters, objects, and shadows should respond coherently. Controllable re‑lighting thus becomes an entry point for exploring the physical expressive capability of video models.

The University of Chinese Academy of Sciences, Cardiff University in the UK, and the Keling AI team jointly proposed JointLight [1], a controllable re‑lighting method for human videos. Given an input video and target ambient light, the model adjusts lighting while preserving the person’s appearance, motion, and scene structure as much as possible. It incorporates geometric priors into generation, allowing RGB and irradiance to be generated together and mutually corrected, supporting control over lighting direction, intensity, and color, providing a way to achieve controllable physical expression in video models. The related work was published at SIGGRAPH Asia 2026.

• Paper Title: JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling

• Project homepage:http://www.geometrylearning.com/JointLight

• Code link:https://github.com/IGLICT/JointLight

Figure 1: Results of the same video under different target lighting conditions. The left column is the input, the middle and right columns are the relighting results; the small sphere represents ambient light, and the small image shows irradiance results.

JointLight supports static and dynamic target ambient lighting, and after fine‑tuning the relighting task using only synthetic data, it can be applied to real videos; in the paper’s dynamic lighting evaluation, all three quality metrics outperform the compared methods. As shown in Figure 1, whether it is nighttime streetlights or afternoon sunlight, both portraits and scenes achieve accurate and controllable lighting rendering results.

Part 1 Research Background

Re‑lighting requires the model to understand the relationships among light sources, geometry, materials, and occlusion: when illumination changes, the brightness and shadows of characters and scenes should change accordingly while preserving the original appearance and motion. This is also a crucial capability for video models to represent the physical world. Traditional rendering relies on accurate geometry and material information, but ordinary videos rarely provide complete 3D structure, and reconstructing complex characters and scenes incurs high acquisition costs.

Diffusion models lower the input barrier for re‑lighting. Methods such as IC‑Light [2] demonstrate the potential of generative lighting editing, but relying solely on text or ambient light conditions makes precise control of light direction and shadows difficult. DiffusionRenderer [3] and UniRelight [4] further support environment‑map‑driven re‑lighting, yet they still require the network to implicitly infer how light interacts with the scene; when faced with complex occlusions and human motion, local lighting relationships may deviate from the target illumination, and the relatively simple synthetic training scenes also limit generalization to real‑world scenes.

Methods such as DiLightNet [5] and PractiLight [6] introduce lighting cues from coarse geometric rendering, but depth errors and boundary gaps cause shading inconsistencies, and using them directly as control conditions may propagate artifacts into the result. Video also needs to maintain cross‑frame lighting coherence; per‑frame geometric errors further increase the risk of flickering. Therefore, the key is to leverage the reliable illumination information provided by geometry while allowing the model to correct unreliable details—this is precisely the motivation behind JointLight.

Part 2 Algorithm Principles

The core idea of JointLight is to combine the illumination basis of physical rendering with the generative capability of video models. The entire framework revolves around two key designs: first converting the target environment lighting into irradiance cues aligned with the image plane, and then using asynchronous joint modeling to let the image and cues mutually refine each other during generation.

Provide lighting cues using geometric priors

Lighting control first needs to answer “where should be bright and where should be dark.” JointLight records the color and intensity of light from all surrounding directions using an HDR environment map, then estimates a monocular point cloud from each input video frame and converts it to a mesh, discarding triangle faces that cross depth-discontinuous regions. Afterwards, the mesh is assigned a uniform white diffuse material and rendered under the target environment lighting with Mitsuba 3 [9], producing a coarse irradiance map aligned with the image plane. However, the constructed coarse irradiance contains many flaws, and using it directly as a lighting control condition can easily introduce errors.

Figure 2: Comparison of RGB, GT irradiance and rough irradiance. Geometric errors cause the rough irradiance to deviate from the true light‑dark relationship.

As shown in Figure 2, compared with the true irradiance, the coarse irradiance loses details such as hair strands, clothing folds, and leaves; the character edges exhibit black gaps, and there are local shading deviations. If these results are used directly as fixed control conditions, the model may carry the gaps and incorrect shading into the generated images, resulting in black borders, fake shadows, or loss of detail. In video, frame‑by‑frame geometric errors can also cause flickering of light and shadow. Therefore, JointLight treats the coarse irradiance as a correctable cue, while using the true irradiance as the generation target during actual training.

Generate Video and Irradiance Together

JointLight fine‑tunes the Wan2.2‑TI2V‑5B [7] video generation model, feeding the input video, noisy RGB video, and noisy irradiance video into the same diffusion Transformer. The model simultaneously generates re‑illuminated frames and refines the lighting distribution; the two modalities correspond spatially and temporally and exchange information within a single model.

Figure 3: Joint training process. RGB and irradiance use independent noise timesteps and exchange information within the same model.

Here, “asynchronous” means that RGB and irradiance can be at different denoising stages. During training, we sample noise timesteps for the two modalities separately, allowing the model to learn to cooperate even when the image and lighting cues have different levels of clarity. In this way, irradiance does not have to always serve as a completely reliable conditioning map; it can participate in generation and be gradually corrected.

If we directly use coarse irradiance as a condition, it is difficult to accurately reconstruct the target lighting; and merely adding joint modeling of RGB and irradiance is also insufficient to eliminate the gap between coarse cues and true irradiance. To adapt to real videos, the training also incorporates domain-matching techniques: in the second stage, noisy coarse cues randomly replace the noisy ground truth, allowing the model to encounter geometric errors during inference early. Adding an appropriate amount of noise can reduce high-frequency artifacts while preserving large-scale light‑dark relationships; joint modeling enables the model to supplement and correct local details based on the input content and target illumination.

Figure 4: Inference pipeline. After estimating geometry, rendering irradiance and adding noise, re‑lighting results are generated via asynchronous sampling.

During inference, the noisy irradiance participates in guidance from the start, but is kept fixed initially and denoised by the RGB branch; after a preset number of steps, both are denoised together. This allows the model to effectively exploit the low‑frequency illumination distribution in the coarse irradiance cues under geometric priors, while adding noise mitigates the adverse effects of high‑frequency artifacts.

High‑quality synthetic data aids Sim2Real transfer generalization

The team used Unreal Engine 5 [10] to keep the scene, characters, and actions fixed, only changing the lighting, simultaneously rendering RGB and ground‑truth irradiance to obtain paired data that are difficult to capture in real photography. Realistic visuals and diverse scenes help narrow the gap between synthetic and real domains.

The dataset contains 6,800 dynamic scenes, 68,000 video clips, covering 30 indoor and outdoor environments, 60 characters, 600 body animations, 20 expressions, and 821 HDR environment maps. Each scene uses 10 lighting setups, with 20% featuring rotating lights. The diverse combinations enable the model to learn a wider range of lighting relationships.

Based on this pipeline, we also constructed two test benchmarks, SynthStatic and SynthDynamic, which use fixed and rotating target environment lighting respectively, providing paired ground truth for re‑lighting dynamic scenes, used to assess lighting control accuracy and generation quality.

Figure 5: Diverse synthetic scenes provide paired RGB and irradiance, supporting transfer to real videos.

Part 3 Results Showcase

Static Lighting: Balancing Lighting Accuracy and Visual Fidelity

Relighting not only needs to respond to target illumination but also preserve the identity of the person, clothing texture, and scene content. DiffusionRenderer and UniRelight can better respond to lighting direction and color, but in some complex scenes they still exhibit skin and clothing texture distortion or inaccurate local lighting. JointLight clarifies the lighting relationships in the scene through irradiance cues, and while adjusting illumination it better retains the original appearance. In the SynthStatic test, its PSNR reaches 16.62 dB, LPIPS is 0.251, both the best among the compared methods; SSIM is 0.656, slightly lower than UniRelight’s 0.661.

Figure 6: Static lighting comparison.

To further verify the transfer effect, we performed static image evaluation on the real captured HumanOLAT [8] dataset, which was not used in training. JointLight’s PSNR, SSIM, and LPIPS are 23.89 dB, 0.928 and 0.078 respectively, all of which are the best among the compared methods.

Table 1: HumanOLAT static image relighting results.

Dynamic Lighting: Continuously Adjust Shadows with Target Lighting

When character motion and light source changes occur simultaneously, the model needs to continuously update lighting direction, shadows, and local illumination, increasing the difficulty of control. The experiment rotates the target ambient light to test whether the generated results can respond coherently to lighting changes. In the SynthDynamic test, JointLight achieves PSNR, SSIM, and LPIPS of 16.39 dB, 0.631, and 0.263 respectively, all three metrics surpassing the compared methods. Compared with UniRelight, PSNR improves by 1.49 dB and LPIPS decreases by 0.040. Combined with video comparisons, this shows that explicit irradiance guidance helps the model maintain controllability when the target lighting varies over time.

Figure 7: Dynamic lighting comparison.

Table 2: SynthDynamic dynamic lighting test results.

More results

Irradiance not only provides a basis for lighting control, but also helps observe how the model adjusts the illumination distribution of the scene. By asynchronously jointly modeling pixels and irradiance, the model can produce refined irradiance while generating relighting videos.

Figure 8: Shading illustration. From left to right: input, relighting result, irradiance map, and target ambient light.

Explicit lighting control integrated with geometric priors makes the video lighting more precise and controllable, and more consistent with real-world behavior, especially under direct illumination such as sunlight.

Figure 9: More results.

Part 4 Conclusion

Video generation is shifting from pursuing visual realism of frames to controllable expression of physical relationships in scenes. Lighting is the crucial link between 3D scenes and 2D images: it reveals object shape and material, and determines how characters and environments visually blend together. To let generative models adjust light and shadow according to the creator’s intent, the relationships among light sources, geometry, and occlusion must be incorporated into the generation process. JointLight constructs illumination cues using geometric priors, and through asynchronous RGB and irradiance joint modeling and domain matching, it enables the model to correct lighting cues while generating frames, combining the interpretability of physical rendering with the expressive power of video generation, thus providing a pathway for precise control of light and shadow in dynamic scenes.

Given an input video and target ambient lighting, JointLight can reorganize the illumination and shadows of the characters and surrounding scene, allowing the final footage to retain creative flexibility for lighting adjustments. This provides a more versatile light‑and‑shadow editing approach for film post‑production, digital human rendering, and virtual content creation, and demonstrates the value of geometric priors in generative modeling: even when 3D information is imperfect, it can still serve as an effective constraint for generation and guide physical relationships. As geometry, material, and light transport become more deeply integrated into the generative process, video models are expected to support richer scene editing, extending creators’ control from visual appearance to the physical factors that form the image.

References

[1] Henglei Lv, Bailin Deng, Xiaoqiang Liu, Yichen Wang, Haoxian Zhang, Pengfei Wan, Lin Gao*. JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling. SIGGRAPH Asia, 2026.

[2] Lvmin Zhang, Anyi Rao, Maneesh Agrawala. Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport. ICLR, 2025.

[3] Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Chih-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, Zian Wang. DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models. CVPR, 2025: 26069–26080.

[4] Kai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang. UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting. arXiv:2506.15673, 2025.

[5] Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, Xin Tong. DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation. SIGGRAPH, 2024: 1–12.

[6] Yotam Erel, Rishabh Dabral, Vladislav Golyanik, Amit H. Bermano, Christian Theobalt. PractiLight: Practical Light Control Using Foundational Diffusion Models. ACM TOG, 44(6), 2025: 1–11.

[7] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, Ziyu Liu. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

[8] Timo Teufel, Pulkit Gera, Xilong Zhou, Umar Iqbal, Pramod Rao, Jan Kautz, Vladislav Golyanik, Christian Theobalt. HumanOLAT: A Large-Scale Dataset for Full-Body Human Relighting and Novel-View Synthesis. ICCV, 2025: 29131–29141.

[9] Wenzel Jakob and Sébastien Speierer and Nicolas Roussel and Merlin Nimier-David and Delio Vicini and Tizian Zeltner and Baptiste Nicolet and Miguel Crespo and Vincent Leroy and Ziyi Zhang. Mitsuba 3 renderer. https://mitsuba-renderer.org.

[10] Unreal Engine 5. https://www.unrealengine.com/unreal-engine-5.

© THE END 

For reprints, please contact this public account for permission

Submit articles or request coverage: liyazhou@jiqizhixin.com